# LLM Observability: A Technical Deep Dive How do you debug an LLM application when the answer is wrong, latency suddenly spikes, or token costs explode? Traditional application monitoring isn't enough. LLM applications require a new layer of observability that captures prompts, model calls, agent workflows, retrieval steps, token usage, and much more. ## Introduction Modern LLM applications are no longer simple request-response APIs. A single user query may involve: * API Gateway * Authentication * Multiple LLM calls * Retrieval from a Vector Database * Tool execution * Agent planning * Memory lookup * Reranking * Response generation If any step fails, understanding **where**, **why**, and **how** becomes difficult. This is exactly what **LLM Observability** solves. Unlike traditional monitoring that focuses on CPU or HTTP latency, LLM observability helps answer questions like: * Why did the model hallucinate? * Which retrieval step failed? * Which tool took 8 seconds? * Why did token usage suddenly double? * Which model version produced poor answers? * Where is most of the latency coming from? ![](https://cdn.hashnode.com/uploads/covers/639ca8d6052fc36d4a8f42ec/e5970548-c7a2-40de-881d-d3c697f5bd01.png align="center") ## The Three Pillars of Observability Every observability platform—whether for distributed systems or LLM applications—is built on three core pillars: 1. Metrics 2. Logs 3. Traces Together they provide answers to three different questions: | Pillar | Answers | | --- | --- | | Metrics | What is happening? | | Logs | What exactly happened? | | Traces | Where did it happen? | Think of them as progressively increasing levels of detail. ### 1\. Metrics Metrics are numerical measurements collected over time. They provide a high-level view of system health. Examples in traditional systems: * CPU utilization * Memory usage * Request latency * Error rate * Requests per second For LLM systems, metrics become even more interesting. Examples: * Prompt tokens * Completion tokens * Total cost * Requests/minute * Tool execution latency * Cache hit ratio * Retrieval latency * Hallucination rate (if measured) * User satisfaction score * Agent step count Metrics are lightweight and ideal for dashboards and alerting. Example dashboard: ```plaintext Average latency: 1.8 sec P95 latency: 3.7 sec Average tokens/request: 1842 Error rate: 1.2% Daily cost: $146 ``` Metrics answer: > "Is something wrong?" They usually **do not explain why** ### 2\. Logs Logs are detailed records of individual events. Unlike metrics, logs preserve context. For an LLM application, logs might include: ```plaintext { "request_id": "abc123", "model": "gpt-4.1", "temperature": 0.2, "prompt": "...", "completion": "...", "input_tokens": 1452, "output_tokens": 321, "latency_ms": 1832 } ``` Logs help investigate questions like: * What prompt was actually sent? * What response came back? * Which model version was used? * Which user triggered this? * Which tool failed? * What exception occurred? Good logs should be: * Structured * Searchable * Correlated * Timestamped Prefer structured logs: ```plaintext { "event":"tool_execution", "tool":"weather_api", "latency":482, "status":"success" } ``` Structured logs make querying much easier. ### 3\. Traces Metrics tell you **something is wrong**. Logs tell you **what happened**. Traces tell you **where it happened**. Tracing follows a request from beginning to end as it travels across multiple services. Imagine a user asks: > "Summarize my quarterly sales." The request may flow like this: ```plaintext Gateway ↓ Authentication ↓ Planner Agent ↓ Retriever ↓ Vector Database ↓ LLM ↓ SQL Tool ↓ LLM ↓ Response ``` A trace records every one of these steps. This allows engineers to identify: * Which component was slow * Which component failed * How requests propagated * Total end-to-end latency Without traces, debugging distributed AI systems becomes guesswork. ## Understanding Traces and Spans Tracing revolves around two fundamental concepts: * Trace * Span These are often confused. Let's break them down. ### What is a Trace? A trace represents the **entire lifecycle of one request**. One user request equals one trace. Example: ```plaintext User asks: "What were my sales this quarter?" ``` Everything that happens afterward belongs to a single trace. ```plaintext Trace ├── Authenticate ├── Retrieve documents ├── LLM call ├── SQL query ├── Final response ``` Each trace has a unique Trace ID. Every service involved shares this ID. ### What is a Span? A span represents **one operation inside a trace**. Think of spans as building blocks. Example: ```plaintext Trace ├── Span: API Gateway 22 ms ├── Span: Authentication 41 ms ├── Span: Vector Search 182 ms ├── Span: LLM Call 2150 ms ├── Span: SQL Query 98 ms ├── Span: Final Formatting 12 ms ``` Each span contains metadata such as: * Start time * End time * Duration * Status * Attributes * Events * Parent span Because spans have parent-child relationships, the backend can reconstruct the complete execution tree ![](https://cdn.hashnode.com/uploads/covers/639ca8d6052fc36d4a8f42ec/40ac7683-5225-4910-9a7d-62220650474c.png align="center") ## OpenTelemetry (OTel) Collecting telemetry separately for every vendor quickly becomes painful. Imagine rewriting instrumentation every time you move from Datadog to Grafana, or from Jaeger to another backend. OpenTelemetry solves this problem. OpenTelemetry (OTel) is the **Cloud Native Computing Foundation (CNCF)** open standard for generating, collecting, and exporting telemetry data. It is vendor-neutral and supported by most major observability platforms. It standardizes three signal types: * Metrics * Logs * Traces Instead of instrumenting your application differently for every backend, you instrument once using OpenTelemetry. The backend can change later without changing application code. This vendor-neutral approach prevents vendor lock-in while enabling interoperability across observability ecosystems ![](https://cdn.hashnode.com/uploads/covers/639ca8d6052fc36d4a8f42ec/c06d43da-81ed-4711-b4be-f3e2c5ca0489.png align="center") ## The OpenTelemetry Observability Pipeline One of the biggest strengths of OpenTelemetry is its decoupled architecture. Instead of applications sending telemetry directly to an observability platform, data flows through a standardized pipeline. ```plaintext Application ↓ OpenTelemetry SDK ↓ OpenTelemetry Collector ↓ Observability Backend ``` Let's understand each component. ### 1\. OpenTelemetry SDK The SDK runs inside your application. Its responsibilities include: * Creating spans * Recording metrics * Capturing logs * Adding metadata * Propagating Trace IDs * Exporting telemetry Example (Python): ```python from opentelemetry import trace tracer = trace.get_tracer(__name__) with tracer.start_as_current_span("llm_call"): response = client.responses.create(...) ``` The SDK should do as little processing as possible to minimize application overhead. ### 2\. OpenTelemetry Collector The Collector is the heart of the pipeline. Instead of every application talking directly to Datadog, Grafana, or Jaeger, they all send telemetry to the Collector. The Collector can: * Receive telemetry from many services * Batch requests * Filter sensitive data * Enrich telemetry * Sample traces * Transform attributes * Route data to multiple destinations * Retry failed exports * Buffer during backend outages For example: ```plaintext 20 Services ↓ OTel Collector ↓ Grafana ↓ Jaeger ↓ S3 Archive ``` A single Collector can fan out telemetry to multiple backends simultaneously, making migrations and multi-tool deployments much easier. ### 3\. Observability Backend The backend stores and visualizes telemetry. Popular options include: * Jaeger * Grafana Tempo * SigNoz * Datadog * Honeycomb * New Relic * Elastic Observability The backend provides: * Search * Dashboards * Trace visualization * Service dependency graphs * Alerts * Cost analysis * Root cause analysis This is where engineers spend most of their debugging time. ![](https://cdn.hashnode.com/uploads/covers/639ca8d6052fc36d4a8f42ec/5598535e-e0d6-44a5-b727-c82388f998fd.png align="center")