A developer who planned to ship a paid Claude Code plugin to mitigate two common user complaints has scrapped the product after rigorous testing revealed the model already addresses the issues. The plugin, designed to filter verbose test output and verify completion claims, proved redundant across three public repositories. The author, alrulop, concluded that Claude Code 2.1.316 running claude-sonnet-5-5 was already self-regulating token usage and accurately reporting task status, rendering the external intervention unnecessary.

The Myth of Raw Token Bloat

The first hypothesis was that raw test output consumes excessive tokens, potentially hitting usage limits or degrading performance. While a verbose test run with a single failure did generate between 10,000 and 18,000 estimated tokens, the model never encountered this raw data in practice. In six logged sessions, Claude consistently piped test commands through tail or grep filters before processing the results. Each filtered command returned only 369 to 931 characters, meaning the model was already applying the exact optimization the plugin intended to provide. The author noted that this behavior occurred both with and without the plugin active, suggesting it is intrinsic to the model’s current instruction set or default configuration.

Completion Claims Held Up Under Scrutiny

The second complaint addressed was the phenomenon of Claude declaring a task "done" while failing to run tests or leaving errors unresolved. To test this, the developer ran 16 sessions across Express, Flask, and Cobra repositories, using a JSON schema to force explicit completion and test-passing reports. An independent checker script verified these claims by running the full suite and grepping for new code. In all 16 valid sessions, the checker agreed with Claude’s status report. There were zero instances where the model claimed success while the suite was failing. While the author cautioned that this is "no disagreement observed" rather than proof of bug-free code, it strongly suggests the model is reliable in its self-assessment under these specific conditions.

Hidden Dangers in Naive Filtering

The investigation revealed that simple filtering strategies can be dangerous. The documentation’s example hook, which uses grep and head, can accidentally hide failures. For instance, Mocha’s output format did not match the grep pattern, causing a failing Express test to return empty results. Furthermore, without pipefail enabled, the exit status of the pipeline often reflects the last command (head or tail) rather than the test runner, leading to false positives for success. The author’s safer approach involves capturing the runner’s status separately from the log output, ensuring that the exit code accurately reflects the test suite’s outcome.

Key Takeaways

  • Claude Code 2.1.316 natively trims test output using tail and grep, reducing token usage by 95-99% without plugins.
  • The model’s self-reported completion status matched independent checker scripts in 16 out of 16 valid sessions.
  • Naive filtering hooks can hide test failures and misreport exit codes if not carefully implemented with pipefail.
  • Cache reads dominate token counts but are billed at a discount; cache writes correlate more closely with actual session cost.
  • The author’s "benchkit" is now public, allowing the community to reproduce these findings and test other models.

The Bottom Line

This is a cautionary tale for the plugin ecosystem: don't build solutions for problems the base model has already solved. The data suggests that Claude’s current behavior is sufficiently optimized for token efficiency and accuracy, making external filters redundant and potentially risky if implemented poorly.