The promise of autonomous agents often dies in the demo phase, but one developer just proved they can actually do the grunt work. Using Pi subagents, a builder tested LessCode, a JavaScript and TypeScript review tool, against real pull requests sourced directly from GitHub. The workflow didn't just run tests; it orchestrated a full evaluation pipeline where one agent found the test cases and others executed them. This isn't just a hackathon win; it's a blueprint for how we should be testing AI-driven developer tools in production.

Orchestrating the Software Factory

The core of this experiment was the 'software factory' concept, where a Pi coordinator agent managed the evaluation by assigning tasks to specialized subagents. One subagent was tasked with sourcing, scouring the web for TypeScript repositories and identifying candidate pull requests that represented valid test cases. Simultaneously, other subagents were assigned the execution role, running LessCode against the specific changes identified by the sourcing agent. This division of labor allowed the developer to bypass the manual tedium of finding and setting up test environments, a task that typically consumes hours of engineering time.

Sandboxed Execution on Proxmox

To ensure the integrity of the evaluation, the entire agent workflow was isolated within a dedicated virtual machine on a Proxmox home lab. The coordinator and its subagents operated strictly within this VM, separated from the rest of the developer's infrastructure. This setup provided a clean slate for each evaluation run and ensured that the assignments given to each runner were narrow and traceable. By isolating the environment, the developer could map specific results back to specific code changes, creating a clear audit trail for the tool's performance.

Catching Hallucinations in Real Time

The real value emerged when the agents encountered a tricky JavaScript refactor in the is-number library. LessCode’s engine, CLI, and MCP interface all failed to propose a patch initially, but the generated reviews contained conflicting claims about typeof checks throwing ReferenceErrors. When the developer manually verified the functions in isolation, the code returned false for the tested inputs, exposing a hallucination in the AI's analysis. This discrepancy triggered a specific improvement cycle: the developer added verification checks to LessCode, allowing it to validate claims against evidence in the source code.

Rerunning for Verification

After patching LessCode to include evidence-based verification, the developer reran the exact same revision of the is-number change. The updated CLI and MCP reports correctly described the behavior for those inputs, and the CLI successfully withheld two claims it could not support with citations. This closed-loop testing demonstrates the power of using subagents not just for generation, but for rigorous regression testing. The ability to pinpoint a failure, fix the underlying logic, and re-verify against the same historical data is what separates toy agents from production-ready systems.

Key Takeaways

  • Pi subagents can effectively divide labor between sourcing test cases and executing evaluations, reducing manual overhead.
  • Isolated VM environments are critical for maintaining traceability and security when running autonomous agent workflows.
  • AI code reviewers are prone to hallucinating errors (like ReferenceError in typeof checks) unless explicitly instructed to verify claims against source evidence.
  • The 'software factory' pattern allows developers to treat agent workflows as reproducible units of work, enabling true regression testing for AI tools.

The Bottom Line

Autonomous agents are finally moving beyond simple task execution into rigorous quality assurance, proving that subagent orchestration is the missing link for reliable AI development.