What is LLM observability?

LLM observability is the practice of recording and monitoring what a large language model application does in production. It captures each prompt and completion, token usage and cost, latency including time to first token, tool calls and errors, and quality signals such as user feedback or evaluator scores, linked together in a trace so that any answer can be explained and problems can be found early.

What is the difference between LLM observability and LLM evaluation?

Observability records what happened in production continuously; evaluation measures quality against a defined test set, usually before a release. They feed each other: traces flagged in production become new evaluation cases, and evaluation scores can be computed on sampled production traces. Observability tells you something went wrong; evaluation tells you whether a change fixed it.

What should you log in an LLM application?

Log the full prompt as sent, the completion as received, model name and parameters, input and output token counts, cost, latency and time to first token, tool calls with their results and errors, and any user feedback. Redact or hash personal data before storage, restrict access to the logs, set a retention period, and decide deliberately whether the logs may leave your infrastructure.

Which tools are used for LLM observability?

Commonly used options include Langfuse and Arize Phoenix, which are open source, LangSmith, which is a hosted service, and the OpenTelemetry semantic conventions for generative AI, which let LLM traces flow into existing observability backends. Many LLM gateways also emit traces and cost data, and model providers expose usage dashboards for their own APIs.

Glossary

LLM Observability

Q: How does Ethora support LLM observability?

Ethora keeps every agent conversation in per-room transcript memory and in the compliance audit trail with immutable log export to S3, so there is a durable record of what each agent said and when, with configurable retention and per-user erasure. Model-level traces can be collected by a tracing proxy in front of the OpenAI-compatible endpoint the agent uses, and on self-hosted deployments all of it stays in your infrastructure.

Seeing what your LLM application actually did in production: every prompt, completion, tool call, token, millisecond and cost, traced end to end so you can explain a bad answer and catch drift before users do.

General definition

LLM Observability extends classic application observability (logs, metrics, traces) to systems whose core component is a probabilistic model. A conventional service either returns the right response or throws an error; an LLM call can return a fluent, confident and wrong answer with a 200 status. Observability therefore has to capture the content of the interaction, not only its timing, and link each final answer back to the prompt, retrieved documents, tool results and model settings that produced it.

Prompt and completion: the full input as sent (system prompt, history, retrieved context, tool definitions) and the output as received, with model name and parameters
Tokens and cost: input and output token counts and the resulting cost, per call and rolled up per user, feature or tenant
Latency: time to first token (TTFT) for streamed responses and total generation time
Tool calls and errors: tool inputs and results, retries, fallbacks and failures, as spans in one trace across a multi-step agent run
Quality signals: user feedback, scores from an evaluator model, groundedness against sources, and flags for refusals or hallucinations

Tooling has converged on the trace as the unit of analysis. Open-source platforms such as Langfuse and Arize Phoenix, hosted services such as LangSmith, and the OpenTelemetry semantic conventions for generative AI all model an LLM interaction as nested spans with standard attributes for model, tokens and prompts, so traces can flow into an existing observability stack. On top of traces, teams sample a slice of production conversations into evaluation sets, so the test suite keeps pace with real usage rather than a static benchmark.

Prompts and completions are user data. Traces of a support or healthcare assistant contain names, account details and possibly PHI, so LLM observability needs the same controls as any store of personal data: PII redaction or hashing before logging, access control on the trace store, retention limits, and a deliberate choice of where the traces live. Sending every prompt to a third-party tracing service is itself a data transfer that a DPA or BAA has to cover.

In the Ethora ecosystem

Ethora records agent conversations at the messaging layer. Every message an agent receives and sends is part of the room, held in the per-room transcript memory that also serves as the agent’s context, and captured in the compliance audit trail with immutable log export to S3. That gives you a durable, reviewable record of what each agent said to whom and when, with configurable retention and per-user erasure to meet privacy obligations.

Model-level traces (token counts, time to first token, tool spans) are best collected where the inference runs. Because Ethora agents talk to any OpenAI-compatible endpoint, you can place a tracing proxy or gateway in front of the model and keep the traces in your own stack. On a self-hosted or dedicated deployment with a self-hosted LLM, prompts, completions and traces never leave your infrastructure, and the platform ships with Grafana and Prometheus monitoring for the servers themselves.

LLM Observability

General definition

In the Ethora ecosystem

Build AI agents on your own stack

Talk to our team

Pick a time that works for you