Published by Winding AI, Inc., the company that makes Griffin. No vendor paid to be listed.Prices read November 19, 2026 · How this site works
LLM Observability Tools10 tools compared · priced per 100k · read Nov 19See the tools

AI observability tools / Agent observability

Read on November 19, 2026

Agent observability: how to trace an AI agent step by step

Agent observability is LLM tracing stretched over a whole agent run: every model call, every tool call, every handoff and every turn, kept as one trace you can replay. It answers why an agent did something, not only how long a call took. Below: what a good agent trace holds, the names OpenTelemetry gives those steps, and which AI agent observability tools draw the run.

Published November 20, 2026 · Updated November 20, 2026.

What an agent trace should hold

One user request becomes one trace. Inside it, a root span for the agent, and under it a child span for each step: the model call that chose a tool, the tool call with its arguments and result, a retrieval, a handoff to another agent, the next model call. Each LLM span carries the model, the provider, input and output tokens, cached tokens, latency and any error. A multi-turn conversation ties several traces together as a session or thread.

With that tree you can open a failed run and see the step where it went wrong: a tool that returned nothing, a loop of the same call, a model that ignored the tool's answer. Without it you only see a slow or expensive request.

The OpenTelemetry names for agent steps

OpenTelemetry's GenAI semantic conventions, now kept in their own repository and still marked Development, name the spans most tools use:

  • invoke_agent {gen_ai.agent.name} and create_agent for the agent itself;
  • execute_tool {gen_ai.tool.name} for each tool call;
  • a span per model call, with gen_ai.operation.name, gen_ai.provider.name and gen_ai.request.model;
  • gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.usage.cache_read.input_tokens for tokens;
  • gen_ai.conversation.id to join the turns of one conversation, and MCP spans named after the MCP method.

There is no cost attribute. OpenTelemetry records tokens, not dollars, so every tool prices tokens with its own table, and two tools can show different costs for the same run.

AI agent observability tools that draw the run

How each tool shows an agent, from its own docs on November 19, 2026. All but Griffin take OTLP traces, so the OTel names above arrive intact.

ToolAgent viewMCPCostOTLP
LangfuseAgent graphs drawn from nested observations; sessionsMCP tracingPer generationYes
LangSmithRuns for every step; threads link multi-turn sessionsNot statedPer callYes
ArizeAgent Trajectory graph (AX); sessions (Phoenix)Not statedRolled up to trace and projectYes
DatadogAgent decisions and handoffsMCP clients auto-instrumentedPer LLM requestYes
BraintrustNested llm, tool and task spansMCP server tracingChild to parentYes
W&B WeaveMulti-turn timeline with nested tool calls; threadsMCP client and serverPer callYes
OpikAgent graphs (LangGraph, Google ADK); threadsNot statedPer spanYes
LaminarLLM calls, tool use and control flow; sessionsNot statedEvery LLM spanYes
Traceloop@agent and @tool decorators (OpenLLMetry)MCP instrumentationCosts by propertyYes
GriffinNone: one record per call, grouped by use caseNoPer call, by use caseNo

Prices for all of them, per 100,000 spans and for a sample month, are on the pricing data page; the ten main tools are compared card by card on the home page.

LLM tracing vs agent observability

LLM tracing grew out of logging single model calls: a prompt, a response, tokens and latency. That is enough for a chatbot that calls one model once. An agent calls models in a loop and acts between calls, so the useful record is the loop. The tools above all started from tracing and added agent views; a proxy that only sees model calls (Helicone, Portkey) shows the calls but not the tool steps between them unless you group them into sessions yourself.

What to look for

  1. A tree, not a list. Nested spans for tool calls under the model call that chose them.
  2. Sessions or threads for multi-turn agents.
  3. Cost that rolls up from each LLM span to the run (Arize and Braintrust say so in their docs).
  4. OTLP in, so you can switch tools without re-instrumenting.
  5. Evaluations on whole runs, not only single answers: Traceloop, for one, ships evaluators for tool selection and agent goals.
  6. Retention long enough to compare this week's runs with last month's.

Where cost per use case fits

Traces tell you why a run failed. A different question is what each feature of your product spends on AI each day. Griffin answers that one: it records every call's model, tokens, cost and latency, grouped by use case, without spans, and replays real calls on cheaper models with a judge (replay grading) before routing a use case that passed. It doesn't draw agent runs, so teams that need the tree use one of the tools above beside it. Griffin is made by Winding AI, Inc., which publishes this site.

Questions

What is the difference between LLM tracing and agent observability?

LLM tracing records model calls. Agent observability records the whole run around them: which tools the agent picked, what each returned, where it handed off, and how many turns it took, so you can see why a run went wrong, not only which call was slow.

Which AI agent observability tools are open source?

Langfuse (MIT core), Opik, Laminar and OpenLLMetry, Traceloop's SDK (all Apache-2.0) are open source. Arize Phoenix is free to self-host under the Elastic License 2.0.

Does OpenTelemetry record what an agent run costs?

No. The GenAI conventions record input, output and cached tokens per call, but there is no cost attribute, so each tool multiplies tokens by its own price list.

Published by Winding AI, Inc., the company that makes Griffin. Sources: each tool's docs as linked from the data page, and the OpenTelemetry GenAI conventions at github.com/open-telemetry/semantic-conventions-genai, read November 19, 2026.