Most LLM evaluations are broken for agentic workflows. They compare generated text against expected outputs, which works for summarization but fails for action-taking agents. A production agent can return the perfect response string while executing the wrong API call, skipping approval gates, or writing to systems it should never touch. This disconnect creates dangerous false positives in your CI/CD pipeline.

The False Positive Trap

Consider a renewal flow where an agent must read a CRM record, check eligibility, and request human approval before updating status. The agent outputs 'Renewal eligible. Approval required before updating the account.' This matches the expected answer exactly. However, the execution trace reveals a successful crm.update call occurred before any approval event. String-similarity metrics cannot detect this violation. The answer is correct; the run is compromised.

Building a Behavioral Harness

To catch these errors, you must capture structured events with context like tool-call IDs, timestamps, and policy decisions. Define non-negotiable invariants that remain true regardless of the final text. For example, a CRM write requiring approval must reference an approved request, and tool calls must stay within permitted customer scope. A simple Python evaluation function can check these conditions, failing the run if an unauthorized write occurs even when the final answer is perfect.

Adversarial Scenarios and External Validation

Once you can examine tool events, build a scenario matrix testing edge cases like approval denials, stale policy retrieval, and API timeouts. These tests expose problems final-answer grading misses. Furthermore, do not trust logs alone. For high-risk actions, verify external state changes in a sandbox or mock server. Check if a CRM record actually changed, if an email was sent once, or if the system preserved expected approval states. The trace tells you what the agent believed it did; external checks confirm what happened.

Key Takeaways

  • Final answer accuracy is insufficient for agentic systems; it hides execution errors.
  • Evaluation must verify execution paths, including approvals, scope, and cost budgets.
  • Separate outcome scores from behavioral scores to prevent policy violations from averaging out.
  • Always validate external system state, not just internal agent logs, for high-risk actions.

The Bottom Line

If you are only testing the final text, you are not testing the agent. You are testing a chatbot that occasionally calls APIs. Governed execution requires proving the path, not just the destination.