Marc Kumiko’s recent experiment with Kumiko, an app designer platform currently in stealth at studio.kumiko.rocks, offers a sobering look at current LLM capabilities in agentic coding. The test involved five different models attempting to build a maintenance app for a metalworking shop, complete with inspection protocols, PDF exports, and audit trails. While the headline costs ranged from a mere $0.06 to $0.45, the initial conclusion that 'Sonnet wins and Luna is useless' collapsed under scrutiny. The majority of the 'red' test failures were attributed to flaws in the testing infrastructure and prompt ambiguity, not the models' inherent reasoning abilities.

The Illusion of Green

The initial testing framework relied on a 63-point structural check, verifying entities, screens, and navigation. In the first run, Sonnet 5.5 achieved a perfect 63/63 score for $1.16. However, this 'green' status was deceptive. The model had omitted critical business logic: PDF actions only confirmed existence without generating files, out-of-range measurements weren't flagged, and safety locks for continued operation were missing. A maintenance manager would have caught these gaps immediately, proving that structural validation is insufficient for functional correctness. Consequently, the team added seven behavior tests to cover deadlines, limit checks, locks, and PDF generation.

Infrastructure Failures Mask Model Performance

When the five modelsβ€”Sonnet 5.5, gpt-6.1-terra, gpt-6.1-sol, gpt-6-luna, and qwen3.8-27bβ€”were re-evaluated with the stricter behavior tests, the results were chaotic. Sonnet 5.5 cost $0.454 and hit check limits; gpt-6.1-terra cost $0.181 and also hit limits; gpt-6-luna cost $0.056 but failed 0/7 behavior tests. However, deep dives revealed that many failures were self-inflicted by the test rig. The limit check failed for every model because the test asserted undefined instead of notOk, a bug in how the test framework interpreted the spec's derived inRange construct. Furthermore, two boot rules in the generator caused errors that the models couldn't fix because they hadn't written the offending code.

Prompt Ambiguity and the Fix

The test records assumed certain fields were optional, a detail absent from the prompt. This caused gpt-6.1-sol to mark inspector and date as required, breaking every test that needed a protocol. After identifying these issues, the team updated the generator to handle open actions automatically and modified the prompts to explicitly list test record fields. They also enhanced the feedback loop to name the specific spec construct behind failures, such as linking a failed limit check to derived inRange. This transparency allowed the models to correct course more effectively.

Sonnet’s Second Run

With the improved infrastructure and clearer prompts, Sonnet 5.5 was re-tested. It achieved a perfect 63/63 structural score and 7/7 behavior tests across all three steps for a total cost of $0.322. The build step cost $0.247, while the two change requests cost $0.043 and $0.032 respectively. Unlike the first run, there were no boot errors, and the limit check passed on the first submission. However, the author notes that these hints essentially handed the models the answers, raising questions about how well they perform without such specific guidance.

Key Takeaways

  • Structural validation alone is insufficient; 63-point checks passed even when critical business logic like PDF generation and safety locks were completely missing.
  • Test rig flaws and ambiguous prompts caused the majority of initial failures, not model incompetence; fixing the harness and clarifying optional fields resolved most issues.
  • Specific feedback naming spec constructs (e.g., derived inRange) significantly improves model correction rates, though it raises questions about benchmark fairness.

The Bottom Line

This experiment highlights a critical disconnect in current AI agent benchmarks: we are often testing our test harnesses, not the models. Before declaring a model 'stupid' for failing a task, developers must ensure their prompts are unambiguous and their validation logic is robust. The leap from structural correctness to functional reliability remains the primary bottleneck for autonomous app generation.