When a small company says "build us an agent," they often mean a dozen different things. Denys, a Forward Deployed Engineer, refuses to write code until four specific questions are answered: which job is being automated, what defines a passing result, who reviews the output, and what the monthly cost is. This disciplined approach prevents the common pitfall of deploying impressive but useless demos that fail in production.

Selecting the First Job by Failure Mode

Denys argues that the first agent should not be chosen for its wow factor, but for how cheaply it fails. Ideal candidates include drafting weekly reports from internal data or handling initial customer replies where the system can hand off to a human when unsure. Conversely, tasks involving legal weight, contracts, or financial transactions are poor first choices because an agent adds a step rather than removing one. Crucially, source documents must be consistent; if two price lists disagree, the agent will pick one and sound confident, so data hygiene must precede automation.

Defining the Pass Mark Before Development

To avoid scope creep and subjective success criteria, Denys mandates collecting 20 to 30 real-world examples with human-provided answers before building begins. The pass mark must be agreed upon in writing, such as requiring a specific number of correct outputs out of 30, with all errors explicitly flagged as unsure. This metric is then compared against the hours the job currently takes per month, establishing a concrete baseline for ROI. Skipping this step often leads to a post-demo debate where success is defined by whoever is least satisfied.

Three-Tier Review System Based on Risk

Review protocols are dictated by the cost of a mistake. For low-risk internal summaries, a person samples the output weekly. For records feeding reports, the agent flags uncertain items for human review, while a sample of the rest is checked manually. For high-stakes items like money or contracts, the agent only prepares the work, requiring full human approval. A critical test for any agent is its ability to say "I am unsure." If it cannot determine when to hand over control, it is not ready for autonomous operation.

Hard Costs and Silent Failures

Silent failures are more expensive than crashes. Denys cites a personal horror story where a scraping job processed 359 dates in 12 minutes, saved zero rows, and reported success. Now, he enforces checks that verify output was produced and alerts if processes go quiet. The financial math must also be precise. Using API list prices from October 15, 2026, a small workload of 20 tasks a day with 4 model calls each costs between $1.09 (DeepSeek V4.1 Flash) and $62 (Claude Opus 5.5) per month. However, the model is only part of the bill; server costs, tool usage, and proxy traffic (which once cost $3.08 in a single run) must be included. Code-level caps per run are essential to prevent loops from inflating the bill.

Key Takeaways

  • Define the pass mark using 20-30 real examples before writing any code.
  • Choose first jobs with clear inputs/outputs and low cost of error.
  • Implement a three-tier review system based on risk level.
  • Add code-level caps to prevent infinite loops from draining budgets.
  • Verify that agents can explicitly signal uncertainty to trigger human review.

The Bottom Line

Agents are not magic wands but industrial tools that require strict quality control. If a client cannot agree to a defined pass mark and review protocol, the project is not ready for automation, regardless of the demo's flashiness.