Google Agents Cli Observability

ADK Observability Guide

Cloud Trace works out of the box — no infrastructure needed. Prompt-response logging and BigQuery Agent Analytics require Terraform-provisioned infrastructure (service account, GCS bucket, BigQuery dataset). Run agents-cli infra single-project --project PROJECT_ID to provision these resources. See references/cloud-trace-and-logging.md for details, env vars, and verification commands. If your project isn't scaffolded yet, see /google-agents-cli-scaffold first.

Order of operations for agent_runtime deployments

For deployment_target = agent_runtime, run agents-cli infra single-project before the first agents-cli deploy. The Terraform module owns the entire Reasoning Engine resource (display_name, service account, deployment spec, env vars), so applying it after a SDK-based deploy creates a state mismatch — Terraform has no record of the SDK-deployed instance and cannot layer env vars onto it without taking ownership of the whole resource.

If you have already run agents-cli deploy, you have two options:

1. Switch to Terraform-managed. Delete the SDK-deployed Reasoning Engine, then run agents-cli infra single-project followed by agents-cli deploy. Sessions and any in-flight state on the previous instance are lost. 2. Keep the SDK-deployed instance. Skip infra single-project and set the observability env vars on the running instance directly via the vertexai client update API. You will also need to grant the instance's service account the IAM permissions required to emit telemetry — writing to the logs GCS bucket, BigQuery dataset access, log writer, etc. See deployment/terraform/single-project/iam.tf and telemetry.tf in your scaffolded project for the full set of bindings the Terraform module would otherwise provision. Terraform-managed env vars are not available in this mode.

Reference Files

---

Observability Tiers

Choose the right level of observability based on your needs:

Ask the user which tier(s) they need — they can be combined. Cloud Trace is always on; the others are additive.

---

Cloud Trace

ADK uses OpenTelemetry to emit distributed traces. Every agent invocation produces spans that track the full execution flow.

Span Hierarchy

invocation
  └── agent_run (one per agent in the chain)
        ├── call_llm (model request/response)
        └── execute_tool (tool execution)

Setup by Deployment Type

View traces: Cloud Console → Trace → Trace explorer

For detailed setup instructions (Agent Runtime CLI/SDK, Cloud Run, custom deployments), fetch https://adk.dev/integrations/cloud-trace/index.md.

---

Prompt-Response Logging

Captures GenAI interactions (model name, tokens, timing) and exports to GCS (JSONL) and BigQuery (via direct log sinks and external tables). Privacy-preserving by default — only metadata is logged unless explicitly configured otherwise.

Key env var: OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT — OTel GenAI semantic-conventions standard (modes: span_only, event_only, span_and_event, no_content). The scaffolded setup_telemetry() collapses every non-false value to NO_CONTENT (metadata-only); false disables capture. Logging is disabled locally unless LOGS_BUCKET_NAME is set.

For scaffolded project details (Terraform resources, env vars, privacy modes, enabling/disabling, verification commands), see references/cloud-trace-and-logging.md.

For ADK logging docs (log levels, configuration, debugging), fetch https://adk.dev/observability/logging/index.md.

---

BigQuery Agent Analytics Plugin

Optional plugin that logs structured agent events to BigQuery. Enable with --bq-analytics at scaffold time. See references/bigquery-agent-analytics.md for details.

---

Third-Party Integrations

ADK supports several third-party observability platforms. Each uses OpenTelemetry or custom instrumentation to capture agent behavior.

Ask the user which platform they prefer — present the trade-offs and let them choose. For setup details, fetch the relevant ADK docs page from the Deep Dive table below.

---

Troubleshooting

---

Deep Dive: ADK Docs (WebFetch URLs)

For detailed documentation beyond what this skill covers, fetch these pages:

---

Related Skills

• /google-agents-cli-deploy — Deployment targets, CI/CD pipelines, and production workflows
• /google-agents-cli-workflow — Development workflow, coding guidelines, and operational rules
• /google-agents-cli-adk-code — ADK Python API quick reference for writing agent code

BigQuery Agent Analytics Plugin

Opt-in. Enable with --bq-analytics at scaffold time, or add manually to app/agent.py.

An optional plugin that logs structured agent events directly to BigQuery via the Storage Write API. Enables:

• Conversational analytics — session flows, user interaction patterns
• LLM-as-judge evals — structured data for evaluation pipelines
• Custom dashboards — Looker Studio integration
• Tool provenance tracking — LOCAL, MCP, SUB_AGENT, A2A, TRANSFER_AGENT

Enabling

Infrastructure (BigQuery dataset, GCS offloading) is provisioned automatically by Terraform when enabled at scaffold time.

Key Features

• Auto-schema upgrade (new fields added without migration)
• GCS offloading for multimodal content (images, audio)
• Distributed tracing via OpenTelemetry span context
• SQL-queryable event log for all agent interactions

For full schema, SQL query examples, and Looker Studio setup, fetch https://adk.dev/integrations/bigquery-agent-analytics/index.md.

Cloud Trace & Prompt-Response Logging (Scaffolded Projects)

Assumes `/google-agents-cli-scaffold` scaffolding. Observability infrastructure is provisioned by Terraform in scaffolded projects.

Cloud Trace

Always-on distributed tracing via otel_to_cloud=True in the FastAPI app. Tracks requests through LLM calls and tool executions with latency analysis and error visibility.

View traces: Cloud Console → Trace → Trace explorer

No configuration required. Works in local dev (agents-cli playground) and all deployed environments.

Prompt-Response Logging Infrastructure

All provisioned automatically by deployment/terraform/single-project/telemetry.tf (and the cicd/ variant):

• Log sinks — Route GenAI inference logs and feedback logs directly to BigQuery (partitioned tables)
• BigQuery dataset — Telemetry dataset with external tables over GCS data and pre-created log export table
• Pre-created log export table — gen_ai_client_inference_operation_details table with Cloud Logging BQ export schema (labels flattened: dots become underscores)
• GCS logs bucket — Stores completions as NDJSON
• BigQuery connection — Service account for GCS access from BigQuery
• Completions view — Joins BQ log export data with GCS-stored prompt/response data

Check deployment/terraform/single-project/telemetry.tf for exact configuration. IAM bindings grant log sink service accounts roles/bigquery.dataEditor on the telemetry dataset.

Environment Variables

Set automatically by Terraform on the deployed service:

Enabling / Disabling

Enable Locally

Set these before running agents-cli playground:

export LOGS_BUCKET_NAME="your-bucket-name"
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT="NO_CONTENT"

Disable in Deployed Environments

Set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false in deployment/terraform/single-project/service.tf (or the cicd/ variant) and re-apply Terraform.

BigQuery Dataset Naming Convention

BigQuery dataset names cannot contain hyphens. Terraform automatically converts hyphens to underscores when creating dataset names from your project name:

• Project name my-agent → BQ dataset my_agent_telemetry

One dataset is created:

• `{name}_telemetry` — Contains external tables over GCS completions data (NDJSON), the pre-created log export table (gen_ai_client_inference_operation_details), and the completions_view

To discover the actual dataset name in your project:

bq ls --project_id=${PROJECT_ID}

Verifying Telemetry

After deploying, verify prompt-response logging is working:

PROJECT_ID="your-dev-project-id"
PROJECT_NAME="your-app-name"  # The agents-cli project name (not the GCP project ID)

# Check GCS data
gsutil ls gs://${PROJECT_ID}-${PROJECT_NAME}-logs/completions/

# Check BigQuery log export table (logs arrive via sink, may take a few minutes)
bq query --use_legacy_sql=false \
  "SELECT COUNT(*) FROM \`${PROJECT_ID}.${PROJECT_NAME//-/_}_telemetry.gen_ai_client_inference_operation_details\`"

# Query completions external table
bq query --use_legacy_sql=false \
  "SELECT * FROM \`${PROJECT_ID}.${PROJECT_NAME//-/_}_telemetry.completions\` LIMIT 10"

# Query the completions view (joins log export with GCS data)
bq query --use_legacy_sql=false \
  "SELECT * FROM \`${PROJECT_ID}.${PROJECT_NAME//-/_}_telemetry.completions_view\` LIMIT 10"

If data is not appearing: check LOGS_BUCKET_NAME is set, verify SA has storage.objectCreator on the bucket, check application logs for telemetry setup warnings. Log export to BigQuery may take a few minutes to propagate.