Glossary
LLM Evaluation
How you find out whether an LLM application is actually good: benchmarks for choosing a model, task-specific test sets for your own use case, automated judges and human review, run again on every change.
General definition
LLM Evaluation answers two different questions that are often confused. Public benchmarks (MMLU, HumanEval, GPQA, arena-style leaderboards and similar) compare base models against each other on general tasks and help you shortlist a model. Task evaluation measures your application, meaning this model, this system prompt, this retrieval setup and these tools, on the inputs your users actually send. A model that tops a leaderboard can still perform badly in your support chat, so the second kind of evaluation is the one that decides whether you can ship.
- A golden set: representative inputs with reference answers or grading rubrics, drawn from real traffic and edge cases, versioned like code
- Scoring methods: exact or structured match for deterministic outputs, embedding similarity for free text, and LLM-as-judge, where a separate model grades an answer against a rubric
- RAG metrics: faithfulness (is the answer supported by the retrieved passages), answer relevance, and context precision and recall for the retriever itself
- Safety checks: refusals where required, no leakage of personal data, and resistance to a set of prompt injection test cases
- Human review: calibrating the automated judge, resolving disagreements, and grading the cases that matter most
Evaluation earns its keep as a regression suite. Every change to the prompt, model version, chunking strategy or tool set is run against the golden set before release, with scores tracked over time so a quiet degradation shows up as a number rather than as a customer complaint. Production sampling closes the loop: conversations flagged by users or by monitoring are added to the set, which is how evaluation and LLM observability feed each other.
For a support chatbot, a practical suite is a few hundred real questions with expected answers or rubrics; faithfulness and relevance scores for the RAG layer; a hallucination rate measured against the sources; a resolution or escalation-accuracy metric; and a small adversarial set of injection and off-topic prompts. LLM-as-judge makes this affordable, but judge models have known biases (longer answers, first position, their own outputs), so spot-check them against human grades and treat scores as relative, not absolute.
In the Ethora ecosystem
Ethora gives you the raw material for evaluation rather than a scoring product. Every agent conversation is kept in per-room transcript memory and in the audit trail, so real questions and answers can be exported into a golden set, and the one-shot reply endpoint lets you send a single test input to an agent and receive a single answer, which is exactly the call an evaluation harness needs in order to run a suite against a live agent configuration.
Because each agent in the AI SDK has its own system prompt, model and knowledge base, you can evaluate variants side by side: the same golden set against two agents with different prompts, or the same prompt against a cloud model and a self-hosted model reached through an OpenAI-compatible endpoint. Grounding an agent on a curated knowledge base with RAG, and instructing it to say so when the sources are silent, is the change that most often moves a faithfulness score.
