If your recurring scraping jobs are mysteriously corrupting your database, stop debugging your CSS selectors. The problem is likely architectural, not syntactic. Recurring scraping jobs rarely fail because the selector is wrong. They fail because the pipeline cannot tell the difference between a value that changed, a value that was not observed, and a value that was observed badly. Those three cases need different handling. Most implementations collapse them into one: "the new value is different, so overwrite the old one." That is how a temporary timeout becomes a permanent null, and how a layout change quietly corrupts a catalog.

The Three States of Observation

For every field, you must record the observation before deciding what it means. The source material defines three distinct states. "Changed" occurs when the source returned a complete response, the expected element was present, and the normalized value differs from the last accepted value. "Unchanged" happens when the source returned a complete response and the normalized value matches. "Unknown" covers scenarios where the request timed out, returned a partial page, hit an auth wall, or the element was missing. In these unknown cases, the previous value is still the best available value; it is not evidence that the field is now empty. The important rule is simple: an unknown observation must never overwrite a known value.

Implementing a Decision Table

To survive real-world site volatility, you need a strict decision table. If the previous value is known and the current observation is a timeout, partial response, or missing element, the system must keep the previous value, mark it as unknown, and retry. If the previous value is missing and the source is completely absent, do not invent a default; keep it missing. Crucially, if a parser version changes, the system must re-validate a sample before writing any data. A parser deploy can change normalization without the site changing. If you write every row immediately, a bug in the parser looks exactly like a change in the source.

Store Evidence, Not Just Values

A value without provenance is hard to defend later. The article recommends storing specific metadata fields for every record, including observed_at, source_url, http_status, content_hash, parser_version, previous_value, the decision (changed, unchanged, unknown), and the reason (timeout, partial, selector_missing, ok). This structure allows you to answer two vital questions during an incident: when did this value change, and was the source actually observed, or did we simply stop seeing it? Without this audit trail, you are just guessing at data lineage.

Make Unknown Visible

A job can return 200 OK and still learn nothing. You must track two separate metrics: job success rate (did the runner finish?) and observation coverage (for each required field, what percentage of runs produced a complete observation?). Coverage is the metric that catches silent data loss. A scraper with 99% job success and 60% field coverage is not healthy; it is just quiet. Silent data loss is often more dangerous than loud failures because it propagates through your analytics and machine learning models unnoticed.

Key Takeaways

  • Never overwrite a known value with an unknown observation; preserve the last accepted state.
  • Track observation coverage separately from job success rates to catch silent data loss.
  • Store full evidence metadata, including parser version and HTTP status, for every record.
  • Re-validate samples before writing data whenever the parser logic is deployed.

The Bottom Line

Recurring collection gets easier when the pipeline stops pretending it always knows the answer. Separating changed, unchanged, and unknown removes a whole class of data corruption before it reaches the warehouse.