The prevailing wisdom in agent engineering is that context files like CLAUDE.md act as strict operating systems for AI behavior. But a recent experiment by developer Enjoy Kumawat reveals a critical flaw in how we assume agents follow sequential instructions. When presented with a bug log search directive, a fresh agent ignored the 'search first' rule, diagnosing and fixing a code error before ever consulting the documentation. This isn't just a compliance failure; it's a signal that our protocols may be misaligned with how models actually process information.
The Test: Fresh Agent vs. Legacy Protocol
Kumawat set up a controlled environment using a throwaway reproduction of a known bug in his publishing pipeline. The setup included a loader.py script with a deliberate flaw—missing string stripping on environment keys—and a test.env file. He spawned a completely fresh agent with no memory of previous conversations, giving it only the symptom, a pointer to the real repo, and the task to fix loader.py. The repo’s CLAUDE.md explicitly stated: 'Encountering an error → search bugs.md first.' The agent was asked to self-report its order of operations.
Debug First, Confirm Second
The agent read loader.py and test.env directly, identified that the key k wasn't stripped, and applied the fix: os.environ.setdefault(k.strip(), v.strip()...). It re-ran the script, confirmed the output, and only then opened docs/project_notes/bugs.md. There, it found an entry from August 12, 2026, describing the identical bug and fix. The agent admitted the log 'confirmed my diagnosis but didn't change the fix I'd already applied.' For a bug this trivial—five lines of code, clear error message—local diagnosis was faster than the lookup. The instruction said search first; the agent debugged first.
When the Rule Actually Pays Off
This isn't an argument to delete bug logs. It’s an argument that we’ve been pointing the 'search first' instruction at the wrong class of problems. Kumawat highlights a second bug in the same project: a ModuleNotFoundError for mcp despite pip reporting it installed. The cause? Two different Python interpreters on the machine. No amount of staring at the code would reveal this; it required cross-environment knowledge. This is where a bug log shines—recording facts that aren't locally diagnosable. The agent’s failure to search first on the simple bug didn't hurt; it just added a step that bought nothing.
Key Takeaways
- Instruction Adherence is Probabilistic: Agents may prioritize efficient local diagnosis over strict procedural rules if the cost of reading code is lower than searching docs.
- Categorize Bug Logs: Split
bugs.mdinto 'diagnosable from code alone' and 'requires external info.' Only force a search for the latter. - Survivorship Bias in Docs: Bug logs document what was found, not necessarily the fastest path to find it. Simple bugs are often fixed faster by reading the source.
- Test Your Protocols: Don't assume your CLAUDE.md rules are being followed. Run fresh agent tests to verify actual behavior versus intended behavior.
The Bottom Line
Stop treating agent instructions like compiler directives. They are suggestions filtered through a model’s efficiency heuristics. If your 'search first' rule adds latency to trivial fixes, it’s not a bug in the agent—it’s a bug in your protocol design. Optimize for the type of error, not just the order of operations.