The term "agentic testing" has flooded vendor homepages this year, but the definitions remain dangerously vague. A new breakdown from the Phonebox team clarifies the distinction for mobile developers: true agentic testing means an AI agent runs the test by deciding each next action based on real-time screen observation, rather than following pre-written steps. This is not just a semantic tweak; it fundamentally changes how failures are handled and how tests are authored.
The Critical Distinction
Most teams conflate agentic testing with AI-assisted testing. In AI-assisted workflows, tools like Copilot write or repair Espresso scripts, but the resulting code still executes deterministically. In agentic testing, the agent acts as the decision-maker at runtime. If the UI changes, a script breaks, and a self-healing locator might patch it. An agent, however, adapts its path to reach the goal. Tricentis, Momentic, Katalon, and Autify align on this definition, while Forrester labels the broader category "autonomous testing platforms," avoiding the "agentic" buzzword entirely.
Limitations and Trade-Offs
Agents are not drop-in replacements for your CI gate. The source material highlights four major shortcomings: non-determinism, where two runs take different paths making step-by-step replay difficult; higher cost and latency because every action is a model call; gaps in handling complex gestures like multi-finger rotations; and a lack of inherent business judgment. The agent only knows the rules you explicitly state in the prompt. This makes them poor candidates for strict regression suites but excellent for exploratory passes and reproducing elusive bug reports on new screens.
Implementation on Android
To run agentic tests, the AI needs a device it can read and act on. On a local machine, this requires an emulator paired with a mobile MCP server. For agents running in the cloud, hosted devices work via HTTP transport. The guide provides a specific claude mcp add command using a Phonebox endpoint. The key to success isn't the infrastructure, but the prompting strategy: you must instruct the agent to "read the screen before every action" and demand evidence, such as exact screen text, to make the output verifiable.
Key Takeaways
- Agentic testing is defined by runtime decision-making, not just AI-generated code.
- Non-determinism makes agents unsuitable for strict CI gates but ideal for exploration.
- Complex gestures and business logic judgment remain significant gaps in current agent capabilities.
- Verifiability requires prompting agents to quote screen text or provide recordings as evidence.
The Bottom Line
Stop letting vendors blur the line between autocomplete and autonomy. If your test doesn't decide its own path, it isn't agenticβit's just a script with a fancy editor.