Debugging automation is often a game of assumptions, and one developer’s recent field note highlights a critical failure in that logic. An autonomous Claude Code agent, running daily on a Windows PC to collect open calls from data competitions and government programs, reported that a source had changed its layout. The digest indicated 'structure change suspected, check the parser' after returning zero items. However, the parser was innocent. The real culprit was a bot check that never delivered the page in the first place.

The Lie of the Empty Body

The developer investigated the November 8 run and discovered that the server responded with an HTTP 202 status code and an empty body, accompanied by an 'x-amzn-waf-action: challenge' header. This indicates an AWS Web Application Firewall bot check requiring client-side script execution, which the headless collector did not perform. Consequently, the parser received an empty string and correctly returned zero items, leading the system to falsely diagnose a layout change. The same error hint appeared on November 3 and 7, though the specific response headers for those days were not logged, leaving the cause for those earlier failures unverified.

Fixing the Feedback Loop

To address this, the developer updated the fetch function to raise a specific error when detecting an HTTP 202 with an empty body or the WAF challenge header. This change ensures the digest reports 'bot check page, not a parser problem, can't be collected automatically' rather than misleading the operator into inspecting HTML selectors. The collector now explicitly skips the source for that run, attempting again the next morning. This distinction is vital for maintaining trust in automated digests; without it, operators waste time chasing ghosts in the code while infrastructure barriers go unnoticed.

The Hidden Danger of Error Inheritance

A secondary, more dangerous issue was uncovered during an independent code review. The fetch function is also used by a forum collector that processes posts individually. This collector catches errors using 'except (URLError, TimeoutError, OSError)'. The newly introduced error for bot checks was initially a plain runtime error, which would have bypassed these handlers. This meant that a single post behind a bot check would have crashed the entire run, dropping all links from that forum source for the day. The developer rectified this by subclassing the new error from OSError, ensuring existing handlers could catch it gracefully, and added a test to verify the behavior for this specific caller.

Key Takeaways

  • Always inspect response headers and status codes before blaming parsing logic for empty results.
  • Distinguish between infrastructure blocks (like WAF challenges) and application-level changes (like HTML restructuring) in your error messaging.
  • When adding new error types, audit all callers and their exception handling to prevent cascading failures.
  • Inheriting from existing error families (like OSError) is often the safest way to integrate new failure modes without breaking robust error handling.

The Bottom Line

Blindly trusting your error messages is a rookie mistake in infrastructure. If your scraper says the layout changed, it might just be that the server told you to go away.