Practical guide · Agent engineering

How to observe and debug AI agents

Agent observability connects a user's request to each model call, retrieval step, tool execution, approval, and final state. A useful trace shows what the system did and whether the user's job was completed. It should help you reproduce a failure without turning every prompt, document, or tool argument into permanent log data.

Published November 21, 2026 · 10 minute read

The short answer

Assign one trace ID to the full user task. Nest spans for every agent, model, retrieval, tool, policy, approval, and external service step. Record versions, timing, token use, status, stable resource IDs, and typed errors. Keep the final business outcome beside the trace because a technically successful tool call can still leave the task unfinished. Capture prompt and result content only under a documented access, redaction, retention, and deletion policy.

Start with a trace contract

Write down the units you expect to see before choosing a dashboard. The contract should survive a framework or model change. Google ADK's trace documentation models an agent run as a root span with child spans for workflows, model generations, and tool executions. It exports through OpenTelemetry Protocol, so the same shape can go to any compatible backend.

Minimum trace contract for an AI agent run
SpanRecordQuestion it answers
User taskTrace ID, workflow version, tenant-safe subject ID, final outcomeDid the requested job finish?
Agent or workflowAgent name and version, parent, stop reason, step countWhich control path ran?
Model callProvider, requested and returned model, tokens, latency, finish reasonWhere did time and model spend go?
RetrievalQuery version, data-source IDs, document IDs, ranks, scoresDid the model receive the needed evidence?
Tool callTool version, call ID, normalized argument digest, result ID, typed errorDid the right operation run on the right target?
Policy and approvalPolicy version, decision, reviewer role, wait time, expiryWhy was the action allowed, denied, or delayed?
External serviceDestination class, status, latency, retry count, receipt IDDid a dependency fail or duplicate an effect?

OpenTelemetry's GenAI attribute registry defines common names for agent identity, conversations, operations, model requests, retrieval documents, tools, and evaluation scores. The GenAI conventions have moved to a dedicated repository and some fields are still evolving, so pin the convention version used by your instrumentation rather than treating every current attribute as permanent.

What to capture on every run

Start with metadata that supports grouping and debugging: application release, agent and prompt versions, model name, tool versions, timestamps, statuses, token counts, latency, retries, and final outcome. Carry the trace context into remote tools and services. Otherwise, the waterfall stops exactly where many failures occur.

OpenAI's Agents SDK tracing guide records runner tasks, model turns, agents, generations, function tools, guardrails, and handoffs. It also supports custom processors and explicit flushes for background workers. The provider dashboard is useful, but keeping your own stable task and outcome fields prevents the trace from becoming tied to one SDK's span names.

  • Use stable internal IDs for prompts, policies, datasets, tools, and releases. Human-readable labels can change.
  • Record a typed error and retryability instead of parsing an exception message later.
  • Keep model latency separate from tool and queue latency so one average does not hide the bottleneck.
  • Record whether the task completed, partially completed, asked for clarification, escalated, or failed.
  • Attach evaluation labels after review rather than rewriting the original trace.

A worked support-agent trace

A customer asks why an order is late. The agent finds the order, reads shipment events, drafts an answer, and then offers an unsupported delivery date. The final response is wrong even though every API call returned 200. The trace needs enough structure to show where the unsupported date entered the run.

Worked support-agent trace from request to failed outcome
StepEvidenceFinding
1. RequestOrder ID and delay questionTask and target are clear.
2. LookupOrder found, user authorizedSelection and access are correct.
3. Shipment toolLatest event says delayed; no new estimateThe source contains no delivery date.
4. GenerationResponse promises delivery on FridayThe model added an unsupported fact.
5. Outcome graderGrounding check fails; task marked incorrectAdd a deterministic date-support check and regression case.

This diagnosis should change one component at a time. Add the failed run to the agent test-set workflow, require dates in the answer to match a retrieved field, and compare the same case before and after the fix. A prompt edit without a matched replay does not establish that the failure is gone.

Debug failures from the final state backward

  1. Confirm the final business state in the system of record. Do not trust a success message from the agent.
  2. Find the last span that differs from a successful run with the same task shape.
  3. Check external receipts and tool results before reviewing model text.
  4. Check policy, approval, routing, and retrieval inputs before changing the prompt.
  5. Reproduce with the same versioned inputs, then vary one component.
  6. Turn the failure into a labeled regression case and a production monitor when the pattern is measurable.

Google Cloud's agent observability guide recommends OpenTelemetry instrumentation to connect model reasoning, tool calls, external interactions, cost, and errors. It also describes propagating trace context into remote MCP servers, which keeps a remote tool call attached to the originating agent run.

Protect prompts, tool arguments, and user data

Full content is tempting during an incident and dangerous as a default. Prompts, retrieved documents, model outputs, tool arguments, and results can contain personal data, secrets, customer records, or instructions supplied by an attacker. OpenTelemetry marks several GenAI content attributes as potentially sensitive. OpenAI likewise lets applications exclude sensitive trace data.

  • Collect metadata by default and sample content only for an explicit debugging or evaluation purpose.
  • Redact secrets and fields with known personal data before export, not only in the dashboard.
  • Use opaque stable IDs in spans and keep sensitive payloads in separate storage with narrower access.
  • Set retention and deletion rules by data class. Test deletion against traces, payload storage, and derived datasets.
  • Restrict who can search traces, reveal content, export runs, or attach production examples to eval sets.
  • Log the fact that content was omitted or truncated so investigators do not mistake missing evidence for an empty value.

Build metrics from outcomes, not trace volume

Dashboards should answer release and operating questions. Track task completion and critical policy failures first. Add model and tool error rates, p50 and p95 latency, token and monetary cost per completed task, retry loops, approval rate and wait time, human intervention, and the share of runs with enough telemetry to diagnose.

Slice by workflow, release, model, tool, risk class, and customer-safe cohort. Averages can improve while one expensive tool or high-risk workflow regresses. Connect reviewed production traces to the AI eval release workflow so incidents become test cases and eval changes can be checked against live outcomes.

Agent observability checklist

  • One trace covers the full user task and crosses remote service boundaries.
  • Agent, prompt, policy, model, tool, dataset, and application versions are recorded.
  • Model, retrieval, tool, approval, and external-service work have separate spans.
  • The final business outcome is checked in the system of record.
  • Errors are typed and retries are visible.
  • Content capture has redaction, access, retention, and deletion controls.
  • Metrics are sliced by workflow and release, with alerts tied to user or business impact.
  • Reviewed failures become regression cases with links back to their trace IDs.

Continue learning

Primary sources

  1. Google ADK: Agent activity traces
  2. OpenAI Agents SDK: Tracing
  3. OpenTelemetry: GenAI attribute registry
  4. Google Cloud: Observability for AI agent developers