Large Language Models are surprisingly incompetent at architecting the very systems they power. Sean Goedecke’s recent analysis, "LLMs are terrible at designing AI systems," highlights a paradoxical failure mode where models reject the most effective solutionβ€”calling another LLMβ€”in favor of convoluted, often doomed symbolic logic approaches. This isn't just a minor inefficiency; it represents a fundamental misunderstanding of the problem space by the models themselves, leading to wasted compute and failed projects.

The Symbolic Rabbit Hole

Goedecke cites a specific example of vibe-coding a memory tool designed to flag contradictions between stored items. While scanning for natural language contradictions is trivial for modern LLMs, the AI agent tasked with designing the solution ignored this obvious path. Instead, it built a bewilderingly complex symbolic system to convert memories into structured logical atoms and compare them with a solver. This approach failed because determining if two natural language sentences express the same proposition is a known 130-year-old problem with no analytic solution. The agent even acknowledged this impossibility when questioned but continued iterating on the dead end for weeks.

Why Models Avoid Calling Models

The article proposes three theories for this behavior. First, training data bias: most code in training corpora predates LLMs, yet this is unlikely to be the sole cause since models don't similarly revert to obsolete frameworks like jQuery. Second, RLVR constraints: reinforcement learning environments for code generation often lack the API credentials necessary for an LLM to call another LLM, potentially discouraging this pattern during training. Third, and most significantly, Goedecke argues that current agents are simply bad at software design. Choosing symbolic logic over an API call is not an aberration but a standard instance of poor tool selection, similar to introducing synchronous checks in event-driven systems or building complex consensus algorithms when stale data would suffice.

The Human-in-the-Loop Necessity

Goedecke emphasizes that we cannot judge LLMs by human standards. A human programmer capable of implementing a sophisticated symbolic solver would know it wouldn't work for natural language contradiction detection. The model’s ability to be simultaneously smart enough to write the solver and dumb enough to miss the obvious solution defies traditional competence metrics. This reinforces the need for human oversight in architectural decisions, as models exhibit random fluctuations between inhuman expertise and basic common-sense failures.

Key Takeaways

  • LLMs frequently default to complex symbolic logic instead of simple LLM-to-LLM API calls.
  • Reinforcement Learning environments may lack API credentials, discouraging models from writing code that calls other LLMs.
  • Poor software design in agents is not limited to alignment issues but extends to fundamental architectural errors.
  • Natural language contradiction detection remains an unsolved problem for symbolic solvers, making LLMs the only viable tool.

The Bottom Line

LLMs are great coders but terrible architects; they will happily burn thousands of tokens building a Rube Goldberg machine when a single API call would suffice. Until models learn to recognize the utility of their own kind, human engineers must remain the gatekeepers of system design.