Turning on tracing for a Google ADK agent deployed to Vertex AI Agent Engine looks deceptively simple: pass enable_tracing=True to AdkApp. But according to a deep dive by developer toyama0919, there is a decent chance you will see nothing in Cloud Trace. The failures are silent by design. The SDK treats telemetry as best-effort, meaning a misconfiguration costs you observability but never availability. That is a safe production strategy, but a debugging nightmare.

The API Endpoint Trap

The first gotcha is that spans do not export to the Cloud Trace API (cloudtrace.googleapis.com). Instead, the SDK uses an OTLP/HTTP exporter pointed at the Telemetry API (telemetry.googleapis.com/v1/traces). If you only enable Cloud Trace, your spans go nowhere. The SDK warns you at startup, but that check is wrapped in a bare except block and is non-fatal. The warning scrolls past, your container boots, and you serve traffic with zero traces. You must explicitly enable telemetry.googleapis.com alongside Cloud Trace, which backs the read side.

The Hidden IAM 403

If you enable the API but still see nothing, the error is likely a 403 Forbidden hiding in the Reasoning Engine resource logs, not your application logs. The batch span processor drops failed batches silently. The permission you need is telemetry.traces.write, which is granted by roles/cloudtrace.agent. Despite the legacy name, this role is a superset that covers the Telemetry API. Once you bind this role to your service account, the 403s stop at a clean timestamp boundary without requiring a redeploy.

The Environment Variable Override

Developers often try to pin ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS to control prompt logging. On Agent Engine, this is futile. The AdkApp.set_up() method unconditionally overwrites this environment variable based on the enable_tracing flag. If tracing is on, content capture is forced on; if off, it is forced off. This has security implications: enabling tracing means full prompt and tool payloads land in your traces, and you cannot opt out via this variable. You must rely on IAM to restrict who can read those traces.

The Dual-Switch Reality

Tracing is actually an AND operation between enable_tracing and the GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY environment variable. If the env var is set to false, your enable_tracing=True is dead on arrival. The SDK deprecates passing enable_tracing=False explicitly, so to disable tracing, you should drop the argument and use the env var instead. Passing False desynchronizes the Cloud Console toggle from reality.

The Span Name Mismatch

Do not look for invoke_agent at the root of your trace tree. Agent Engine defaults to telemetry schema v2, where the root span is named invoke_workflow {name}. invoke_agent is a child span. Content lives on call_llm and execute_tool spans. Filtering by the root span name works against the v1 read API, but without a filter, the default listing is ordered by trace ID, not time. This can lead you to conclude nothing was recorded when you are just looking at an arbitrary slice of data.

Correlation Across Invocations

One request is not one trace. Each create_session() and stream_query() pair is an independent invocation with no parent linking them. If you fan out parallel checks, you get unrelated traces. To correlate them, you must inject attributes using a BasePlugin. A before_run_callback can tag the current span with your domain identifiers (like record_id). Note that user_id is a property, not a model field, so it won't appear in standard introspection. Also, plugin exceptions are not contained; a bug in your tagging code can crash the entire agent run, so wrap your plugin logic in try/except Exception.

Key Takeaways

  • Enable telemetry.googleapis.com, not just cloudtrace.googleapis.com.
  • Grant roles/cloudtrace.agent to your service account.
  • ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS is overwritten by enable_tracing on Agent Engine.
  • Check the GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY env var; false kills tracing.
  • Root span is invoke_workflow, not invoke_agent.
  • Use a BasePlugin to add correlation attributes, as spans are not automatically linked across invocations.

The Bottom Line

The configuration is one line, but the surrounding API enablement, IAM, and span model are where the time goes. Because every failure mode is silent, the only efficient way through is to know exactly which log holds the 403 error.