AI Agent Observability: What to Track in Production

Enterprise AI agent adoption splits into two divergent stories. Most organizations can build an agent that works in a demo. Far fewer can run one safely in production. The gap isn't the model or the framework—it's observability. A production agent without trajectory visibility is flying blind. You'll only discover failures when customers do, or when your cloud bill spikes.

This post covers the three observables you must track, the implementation patterns that make them searchable at scale, and how to choose between platforms (Langfuse, Braintrust, Honeycomb) that specialize in this telemetry. It's grounded in what actually breaks in production, not what benchmarks measure.

Why Agent Observability is Not LLM Observability

Standard LLM observability captures a single model call: input tokens, output tokens, latency, cost, which model ran, what the user asked, what the model returned. For a chat application, that's enough—one call, one answer.

An agent is different. An agent makes multiple decisions in sequence: it analyzes the user's request, selects a tool, calls that tool, reads the response, decides whether the response is complete or whether it needs another tool, calls that tool, and repeats. Single-call observability captures none of this sequence. You see the final answer, not the path that led to it.

Worse: a failure at step 2 (tool call fails silently—HTTP 200 with empty data) corrupts every step downstream. By step 5, the error is unrecognizable as originating from one mistake. You need observability that reconstructs the full trajectory—every tool call, memory read/write, decision branch, subagent handoff—linked from first step to last so you can see which step broke and why.

The Three Critical Observables

1. Tool-Call Trajectory

For every tool invocation, capture: which tool was selected (and the reasoning context that led to that selection), arguments passed, response received, latency, cost per call, success or failure.

The silent failure is your biggest threat. A tool returns HTTP 200 (success) but the response body is empty or contains no expected data. The agent interprets this as success and proceeds with corrupted downstream reasoning. In a 5-step workflow, one silent failure at step 2 silently corrupts steps 3, 4, and 5. By the time an end-user notices the output is wrong, the error has propagated through the entire trajectory.

Mitigation: validate that every tool response contains expected fields before using it. Don't trust HTTP status codes alone. Add a circuit breaker that stops retrying after 3 identical failures—infinite retry loops burn through your budget.

2. Memory Operations

Agents often augment reasoning with memory—retrieving past context, storing facts learned during this run, updating a knowledge graph. Capture:

• Retrieval queries (what was the agent asking memory to recall?)
• Retrieval results and relevance scores (did memory return the right context?)
• Write operations (what did the agent store?)
• Relevance metadata (is this memory stale or hallucinated?)

Silent memory failures are common: a retrieval returns the wrong entity (same name, different customer), the agent acts on corrupted context, and downstream steps inherit the error. Memory drift—where retrieval data becomes stale over time—is another failure mode: your agent worked perfectly last week but now refusals are climbing because the knowledge base wasn't updated.

3. Multi-Step Workflow Visibility

Capture the full execution path: every step, decision point, branch taken, subagent handoff, and how they connect. Use trace IDs that survive handoffs between agents—when agent A delegates to agent B, both steps carry the same trace ID so you can reconstruct the full path in your observability platform.

This reveals patterns that metrics alone can't: infinite loops (agent keeps selecting the same tool, getting the same error, retrying), plan drift (agent deviated from its intended path midway), and cascading failures (failure at step 2 triggered a chain of downstream failures in step 4 and 5).

Implementation Patterns for Production Observability

Distributed Tracing with Trace IDs

Assign a unique trace ID to each top-level user request. Pass that trace ID to every agent, tool, and downstream service in the chain. When tracing tools see the same trace ID across disparate systems, they can stitch the full path together—including handoffs to sibling agents, not just hierarchical call stacks.

Structured Logging (JSON, Not Free Text)

Log every meaningful event as a JSON object with a consistent schema:

• timestamp — when did this event occur
• trace_id — the top-level request ID
• agent_id — which agent took this step
• step_id — which step in the trajectory (1, 2, 3...)
• tool_name — which tool was invoked
• success / failure — did it work
• cost — how much did this step cost (in tokens or USD)
• latency_ms — how long did this step take

Free-text logs are not searchable at scale. JSON logs are queryable: "show me all agent A's tool calls to Salesforce that cost over $1 and took more than 5 seconds" is a one-line query. "Find the expensive slow steps" in free text requires humans parsing thousands of lines.

Sampling Strategy: 100% Errors, 5-10% Success

Full tracing on all traffic is expensive. Sample strategically:

• 100% of failed requests (errors, timeouts, low confidence decisions) — you need to catch every failure pattern
• 5-10% of successful requests (random sample) — to detect silent quality drift that metrics don't surface

This balances cost and observability. Most high-volume production systems use this approach because full tracing would triple infrastructure costs for minimal marginal gain.

LLM-as-Judge Trajectory Evaluation

On sampled requests, use a separate LLM (an evaluator model) to score the trajectory quality: did the agent pick the right tool for each step? Was the reasoning sound? Did error recovery make sense? Alert if trajectory quality drifts below baseline—this catches silent degradation that single-metric monitoring misses.

Observability Platforms: Langfuse, Braintrust, Honeycomb

Most production teams don't build this from scratch. Three platforms dominate: Langfuse (LLM-native, agent-specialized), Braintrust (eval-focused, cost tracking), and Honeycomb (broad infrastructure observability). Here's how they compare from an agent-observability angle:

Langfuse specializes in LLM-specific instrumentation and was purpose-built for agent-loop tracing. If you're starting without existing observability, Langfuse or Braintrust are the lower-friction choice. Honeycomb provides the broadest infrastructure observability but requires custom instrumentation for agent-specific signals; it makes sense if you're already using Honeycomb at org scale.

Evaluate based on your existing stack and which observability problem you're solving first. All three work; the question is which one requires the least engineering effort for your specific constraints.

What Observability Actually Prevents

When you can reconstruct the full trajectory of every agent step, you catch failure patterns early:

• Silent tool failures: A tool returns empty data; observability catches it at step 2 instead of step 5
• Infinite loops: Agent retries identically 50 times; you set a hard max of 10 and cap spend
• Quality drift: Agent's reasoning quality degrades weeks into production; LLM-as-judge evaluation surfaces it before customers do
• Cascading errors in multi-agent systems: Agent A hallucinates; Agent B inherits corrupted input; Agent C acts on it—tracing shows the error originated at step 1

Observability as a Production Readiness Signal

Organizations that reach production with agents have observability wired from day one. Organizations that cancel have observability as an afterthought. The difference is not budget or tools—it's that successful teams treat visibility as a requirement, not a luxury. You can't safely scale agents without it.

For broader context on what else separates pilots from production, see our post on why most AI agent pilots never reach production. For deployment patterns that surface these observability gaps early, see how to roll out an AI agent safely.

And for the foundational engineering disciplines behind production agents, see the nine engineering control layers we apply to every system we deploy.