Public benchmarks are increasingly useless for discerning true agent capability, according to a new post from developer Grady. By creating a subjective, highly specific test case—simulating a Steller’s Jay in Three.js—Grady demonstrates that model progress is often obscured by labs optimizing for generic coding and logic tests. The experiment, documented on November 18, 2026, shows that Fable 5.2 and Astra 6.2 have recently crossed the threshold into producing visually coherent, interactive 3D environments that were impossible just months prior.

The Failure of Public Leaderboards

Grady argues that labs optimize agents toward public benchmarks, causing reported capability changes to overstate actual utility for specific users. Public tests are comically overweighted toward verifiable coding and logic tasks, ignoring the 'texture' of quality that subjective benchmarks reveal. When a model passes a standardized test, it doesn’t necessarily mean it can handle the quirky, unstructured demands of a real-world project. The solution? Build your own benchmarks. Grady maintains a personal repository in ~/benchmarks/ to track tasks that fail, using them to gauge the frontier of what is newly possible with each model release.

Fable 5.2 and Astra 6.2 Cross the Threshold

The specific test, STELLAR.md, required the creation of a detailed interactive 3D environment featuring a Steller’s Jay. The agent had to simulate flight, hopping, and passive behaviors with extreme visual fidelity, using only JavaScript and HTML. While early attempts in June 2026 failed on basic interactivity and motion physics, the November 2026 results were decisive. A Fable 5.2 agent, coordinating with an Astra 6.2 agent for visual critique, produced a result that Grady deemed 'solved.' The run took 4.5 hours and cost approximately $80 in API credits, with Fable 5.2 writing the majority of the code while Astra provided the visual reasoning checks.

The Unfair Advantage of Context

Grady admits the test is 'unfair' because his agents have access to previous conversation histories via Pharos, allowing them to learn from past failures. This blurs the line between raw model intelligence and tool-assisted learning. However, Grady argues this reflects reality: agents will always have context. The benchmark isn’t about isolating model weights; it’s about evaluating what is possible on the frontier when history and tools are priced in. The final result wasn’t just a pass; it was a milestone toward a larger vision of AI-generated behavioral simulations.

A Vision for Hyperrealistic Bird Games

The ultimate goal behind this benchmark is a 'Steller’s Jay Simulator'—a VR experience that encodes all known behavioral data about the species into a playable game. Inspired by 'American Truck Simulator' and 'Untitled Goose Game,' Grady wants to use AI to interpret scientific research and video data to emulate the life of a bird. This isn’t just about graphics; it’s about understanding the phenomenological experience of being another creature, a nod to Thomas Nagel’s philosophical work. If Fable 5.2 can handle this, the path to hyperrealistic, AI-driven simulations of other species is open.

Key Takeaways

  • Public benchmarks are misleading because labs optimize for them, overstating general capability.
  • Subjective benchmarks reveal the 'texture' of AI output that pass/fail metrics miss.
  • Fable 5.2 and Astra 6.2 can now produce complex, interactive 3D simulations from natural language prompts.
  • Cross-model collaboration (Fable coding, Astra critiquing) significantly improves output quality.
  • Personal benchmarks should track 'newly possible' tasks rather than just raw intelligence.

The Bottom Line

Stop trusting the leaderboards. If you want to know if an agent can actually help you, build a benchmark that breaks when the model isn't truly capable. Grady’s Steller’s Jay test proves that real utility lies in the messy, subjective details that standardized tests ignore.