LLM Observability: A Technical Deep Dive
observability with OpenTelemetry
How do you debug an LLM application when the answer is wrong, latency suddenly spikes, or token costs explode?
Traditional application monitoring isn't enough. LLM applications require a new layer of observability that captures prompts, model calls, agent workflows, retrieval steps, token usage, and much more.
Introduction
Modern LLM applications are no longer simple request-response APIs.
A single user query may involve:
API Gateway
Authentication
Multiple LLM calls
Retrieval from a Vector Database
Tool execution
Agent planning
Memory lookup
Reranking
Response generation
If any step fails, understanding where, why, and how becomes difficult.
This is exactly what LLM Observability solves.
Unlike traditional monitoring that focuses on CPU or HTTP latency, LLM observability helps answer questions like:
Why did the model hallucinate?
Which retrieval step failed?
Which tool took 8 seconds?
Why did token usage suddenly double?
Which model version produced poor answers?
Where is most of the latency coming from?
The Three Pillars of Observability
Every observability platform—whether for distributed systems or LLM applications—is built on three core pillars:
Metrics
Logs
Traces
Together they provide answers to three different questions:
| Pillar | Answers |
|---|---|
| Metrics | What is happening? |
| Logs | What exactly happened? |
| Traces | Where did it happen? |
Think of them as progressively increasing levels of detail.
1. Metrics
Metrics are numerical measurements collected over time.
They provide a high-level view of system health.
Examples in traditional systems:
CPU utilization
Memory usage
Request latency
Error rate
Requests per second
For LLM systems, metrics become even more interesting.
Examples:
Prompt tokens
Completion tokens
Total cost
Requests/minute
Tool execution latency
Cache hit ratio
Retrieval latency
Hallucination rate (if measured)
User satisfaction score
Agent step count
Metrics are lightweight and ideal for dashboards and alerting.
Example dashboard:
Average latency: 1.8 sec
P95 latency: 3.7 sec
Average tokens/request: 1842
Error rate: 1.2%
Daily cost: $146
Metrics answer:
"Is something wrong?"
They usually do not explain why
2. Logs
Logs are detailed records of individual events.
Unlike metrics, logs preserve context.
For an LLM application, logs might include:
{
"request_id": "abc123",
"model": "gpt-4.1",
"temperature": 0.2,
"prompt": "...",
"completion": "...",
"input_tokens": 1452,
"output_tokens": 321,
"latency_ms": 1832
}
Logs help investigate questions like:
What prompt was actually sent?
What response came back?
Which model version was used?
Which user triggered this?
Which tool failed?
What exception occurred?
Good logs should be:
Structured
Searchable
Correlated
Timestamped
Prefer structured logs:
{
"event":"tool_execution",
"tool":"weather_api",
"latency":482,
"status":"success"
}
Structured logs make querying much easier.
3. Traces
Metrics tell you something is wrong.
Logs tell you what happened.
Traces tell you where it happened.
Tracing follows a request from beginning to end as it travels across multiple services.
Imagine a user asks:
"Summarize my quarterly sales."
The request may flow like this:
Gateway
↓
Authentication
↓
Planner Agent
↓
Retriever
↓
Vector Database
↓
LLM
↓
SQL Tool
↓
LLM
↓
Response
A trace records every one of these steps.
This allows engineers to identify:
Which component was slow
Which component failed
How requests propagated
Total end-to-end latency
Without traces, debugging distributed AI systems becomes guesswork.
Understanding Traces and Spans
Tracing revolves around two fundamental concepts:
Trace
Span
These are often confused.
Let's break them down.
What is a Trace?
A trace represents the entire lifecycle of one request.
One user request equals one trace.
Example:
User asks:
"What were my sales this quarter?"
Everything that happens afterward belongs to a single trace.
Trace
├── Authenticate
├── Retrieve documents
├── LLM call
├── SQL query
├── Final response
Each trace has a unique Trace ID.
Every service involved shares this ID.
What is a Span?
A span represents one operation inside a trace.
Think of spans as building blocks.
Example:
Trace
├── Span: API Gateway
22 ms
├── Span: Authentication
41 ms
├── Span: Vector Search
182 ms
├── Span: LLM Call
2150 ms
├── Span: SQL Query
98 ms
├── Span: Final Formatting
12 ms
Each span contains metadata such as:
Start time
End time
Duration
Status
Attributes
Events
Parent span
Because spans have parent-child relationships, the backend can reconstruct the complete execution tree
OpenTelemetry (OTel)
Collecting telemetry separately for every vendor quickly becomes painful.
Imagine rewriting instrumentation every time you move from Datadog to Grafana, or from Jaeger to another backend.
OpenTelemetry solves this problem.
OpenTelemetry (OTel) is the Cloud Native Computing Foundation (CNCF) open standard for generating, collecting, and exporting telemetry data. It is vendor-neutral and supported by most major observability platforms.
It standardizes three signal types:
Metrics
Logs
Traces
Instead of instrumenting your application differently for every backend, you instrument once using OpenTelemetry.
The backend can change later without changing application code.
This vendor-neutral approach prevents vendor lock-in while enabling interoperability across observability ecosystems
The OpenTelemetry Observability Pipeline
One of the biggest strengths of OpenTelemetry is its decoupled architecture.
Instead of applications sending telemetry directly to an observability platform, data flows through a standardized pipeline.
Application
↓
OpenTelemetry SDK
↓
OpenTelemetry Collector
↓
Observability Backend
Let's understand each component.
1. OpenTelemetry SDK
The SDK runs inside your application.
Its responsibilities include:
Creating spans
Recording metrics
Capturing logs
Adding metadata
Propagating Trace IDs
Exporting telemetry
Example (Python):
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("llm_call"):
response = client.responses.create(...)
The SDK should do as little processing as possible to minimize application overhead.
2. OpenTelemetry Collector
The Collector is the heart of the pipeline.
Instead of every application talking directly to Datadog, Grafana, or Jaeger, they all send telemetry to the Collector.
The Collector can:
Receive telemetry from many services
Batch requests
Filter sensitive data
Enrich telemetry
Sample traces
Transform attributes
Route data to multiple destinations
Retry failed exports
Buffer during backend outages
For example:
20 Services
↓
OTel Collector
↓
Grafana
↓
Jaeger
↓
S3 Archive
A single Collector can fan out telemetry to multiple backends simultaneously, making migrations and multi-tool deployments much easier.
3. Observability Backend
The backend stores and visualizes telemetry.
Popular options include:
Jaeger
Grafana Tempo
SigNoz
Datadog
Honeycomb
New Relic
Elastic Observability
The backend provides:
Search
Dashboards
Trace visualization
Service dependency graphs
Alerts
Cost analysis
Root cause analysis
This is where engineers spend most of their debugging time.
