Glossary

KV Cache

The working memory of a running language model. Every token it has read leaves a key and a value behind, and that store is what makes the next token cheap to produce and long contexts expensive to hold.

General definition

The KV Cache exists because of how the attention mechanism works. For each token, every layer of a transformer computes a key vector and a value vector. When the model generates the next token it attends over the keys and values of all previous tokens. Without a cache those vectors would be recomputed for the entire sequence on every step, which is quadratic work. With the cache they are computed once, stored on the GPU, and reused.

The cost is memory. The cache grows linearly with the number of tokens in the context window and with the model’s layer count and hidden size. A 7-billion-parameter model at 16-bit precision needs on the order of half a megabyte of cache per token, so a 32,000-token conversation occupies roughly 16 GB before the weights themselves are counted. On a shared server the KV cache, not the weights, is usually what limits how many conversations can run at once.

  • Paged attention: vLLM introduced PagedAttention in 2023, storing the cache in fixed-size blocks allocated on demand, like virtual memory, which cut fragmentation and raised throughput
  • Prefix or prompt caching: requests that share the same opening tokens (a long system prompt, a document) reuse one cached prefix instead of recomputing it
  • Smaller caches by design: multi-query and grouped-query attention share keys and values across heads, and the cache itself can be quantized to 8-bit
  • Eviction: sliding-window attention and cache compression drop or compress old tokens to keep memory bounded

Prompt caching offered by hosted model APIs is the same idea sold as a billing feature: a cached prefix is cheaper and faster because its keys and values already exist. The practical rule for application design follows directly. Put stable content first in the prompt and variable content last, so the largest possible prefix is reusable across calls.

In the Ethora ecosystem

Every Ethora agent carries a system prompt and, where configured, retrieved knowledge and a per-room transcript memory. Together these form a long, mostly stable prefix in front of each user message. When the agent runs on a self-hosted model behind vLLM, that prefix is exactly what prefix caching is built to reuse, so a busy support room does not pay the full cost of the system prompt on every reply.

The memory arithmetic also shapes capacity planning for a dedicated deployment. The number of concurrent conversations a GPU can serve is bounded by KV cache memory, which depends on how long you let room transcripts grow before they are summarised or trimmed. Choosing a model with grouped-query attention and a sensible context budget per agent is often a bigger lever on cost than the raw parameter count. The AI SDK lets each agent use a different model, so a lightweight model can handle high-volume rooms while a larger one serves the few that need it.

Get started

Build AI agents on your own stack

Ethora’s AI SDK brings agents, RAG and self-hosted LLMs into your product. Talk to our team.

Start Free
Free tier available Enterprise SLA No vendor lock-in