The hype cycle loves to talk about reasoning, memory, and personality. But for Ariel Chang, the developer behind Eliana, a self-hosted, self-improving security agent, the real battle wasn't the LLM. It was the plumbing. In a recent post-mortem, Chang detailed six specific bugs that consumed weeks of development time, all of which had nothing to do with the model's intelligence and everything to do with how capabilities connected to the real world.
The Silent Failure Epidemic
The first critical error, dubbed "the scanner that scanned, then forgot everything," highlighted a dangerous pattern in agent architecture. The code included full logic to insert scan results into a database, but the database handle was never passed to the capability. Because persistence calls were wrapped in conditional checks (if self._db is not None), the agent successfully ran the scan, parsed the data, and then silently dropped the results. The system reported success, but the database remained empty. It looked like it worked, but it had no memory of its own work. This issue cascaded into a second, more deceptive bug. After fixing the database connection, a scan reported "0 findings." Chang nearly accepted this as a clean result until realizing the SSH channel to the tooling host had never connected. The tool failed instantly, the error was swallowed, and the agent confidently told the user nothing was found. As Chang notes, for an autonomous agent, "no result" and "failed to get a result" must never look the same. If the agent can't distinguish between a clean scan and a broken connection, it will confidently lie to you.
Type Mismatches and Learning Loops
The third bug lived in the seams between libraries. The agent used a standard formula library to compute CVSS scores, which returns a decimal.Decimal object. However, the underlying SQLite database refused to bind this type, causing the entire report-persistence path to crash. It was a one-line float() fix, but it underscored a key truth: bugs don't live inside components; they live in the type mismatches between them. Individual libraries were correct, but their interface was broken. Perhaps the most philosophical bug involved the learning loop. The agent was designed to store assessment experience and recall it later. Storage worked. The recall function existed. But Chang realized that "it learns" is a claim, not a feature, unless you can prove behavior changed. To fix this, Chang implemented a test where sslscan was ranked 5th in tool order during a baseline run. After seeding relevant past experience, the next run recalled the data and moved sslscan to 3rd. Only then was the loop marked CLOSED-LOOP-PROVEN. Without that proof, the system was just a write followed by an unused read.
Runtime and Environment Pitfalls
Infrastructure issues rounded out the list. A systemd service configured with %h to find the user's home directory failed because, in a system-service context, it resolved to root's home instead of the service user's. The binary wasn't there, and the service crashed. This highlighted the gap between "works when I run it by hand" and "works as a managed service." Finally, a de-identification mechanism designed to keep the public repo clean backfired. The system replaced real paths and credentials with placeholders, loading them from environment variables at runtime. But one service's env file was missing keys, causing the agent to fall back to the placeholder values. It dutifully created an empty database file named after the placeholder string, leading to failed lookups against a real but empty database. De-identification is a feature with its own failure modes; a missing key should fail loudly, not fall back silently.
Key Takeaways
- Silent failures are fatal: An agent that reports success when it failed to connect is worse than one that throws an error.
- Prove the loop: Don't claim an agent "learns" unless you can demonstrate a specific behavioral change caused by stored memory.
- Boring is where agents live: The hard part isn't the model; it's the connective tissue between capabilities, dependencies, and runtimes.
The Bottom Line
Stop treating agent development like model tuning. The intelligence is already there; the engineering challenge is the unglamorous, critical work of wiring. If your agent isn't reliable, it's probably not the LLM's faultβit's yours.