Nubank has cracked the code on rapid iteration for customer experience (CX) agents in regulated industries. A new paper from the company’s engineering team details a hypothesis-driven simulation workflow that slashed their update cycles from three weeks to less than five days. By leveraging synthetic customers and simulated tool outputs, the team can screen candidate agents for intent detection and policy compliance without exposing real users to potential failures.

The Snowglobe Simulator at Scale

The core of this methodology is the Snowglobe simulator, which the team deployed against Nubank’s Card Delivery agent and its expanded successor, Card Management. This agent handles the highest volume of chat support in Brazil, making reliability paramount. The simulation allows for multi-step agentic workflows to run without invoking production backends, effectively creating a sandboxed environment where the AI can be stress-tested against complex operational policies and tool usage scenarios.

Metrics That Matter in Production

The results speak to the efficacy of this approach. Across four deployed versions, the simulation scores showed high correlation with production binary evaluator scores. More importantly, in a live A/B test, simulation-guided iteration increased the transactional net promoter score (tNPS) by a massive 36.69 points. The team also screened open-weight configurations across over 16,000 simulated conversations, leading to a model selection that boosted the self-service rate (SSR) by 8.82 percentage points to record highs.

Key Takeaways

  • Simulation allows for safe exploration of models, reasoning settings, and prompts without customer exposure.
  • Synthetic customer reactions enable multi-step workflow testing independent of production backends.
  • High correlation between simulation scores and production metrics validates the pre-deployment screening process.

The Bottom Line

This is how you ship AI agents in 2026: stop gambling with your users' trust and start simulating their reality. If you’re still waiting three weeks to know if your agent broke, you’re already obsolete.