· Daramola Joseph Omoyele
Repair, don't delete: handling errors in price collections
Deleting suspicious price quotes feels safe, but it quietly changes the index. Notes from building PriceLab and Price Quality Explorer.
Every price collection contains mistakes: a price keyed in pence instead of pounds, a zero entered for an item that was out of stock, a seasonal product that disappears for half the year. The instinctive fix is to delete the odd rows. In a matched-item index that choice has consequences.
Why deletion is not neutral
A matched index compares the same items over time. Remove one observation and the item drops out of two comparisons, changing the sample silently. Remove several from one category and that category's movement can shift noticeably.
A more transparent approach
- Diagnose missing values by mechanism. Gaps that recur in the same months each year are seasonal; gaps that hit everything at once are collection failures. They need different treatments.
- Repair clear unit errors. A price exactly one hundred times its item's usual level is almost certainly a unit slip; rescale it and keep a flag.
- Never overwrite the original. Keep raw and analysis values side by side so every decision can be reviewed.
- Write the decisions down. A method note that lists every adjustment makes the result auditable.
Where this applies
The same principles apply to sales data, survey responses and operational records: decide what each error means before you decide what to do with it, and keep a record of what changed.