Glossary

Attention Mechanism

The core operation of a transformer. Attention lets each word in a sequence look at every other word and decide how much it matters, which is how a language model keeps track of meaning across a long passage.

General definition

The Attention Mechanism was introduced for machine translation by Bahdanau, Cho and Bengio in 2014 and became the whole architecture in the 2017 paper “Attention Is All You Need” by Vaswani and colleagues, which defined the transformer that every modern LLM is built on. The idea is simple: instead of compressing a sentence into one fixed vector, let each position compute a weighted mix of all positions, with weights learned from the data.

Each token’s vector (its contextual embedding) is projected into three roles: a query (what am I looking for), a key (what do I contain) and a value (what do I pass on). The attention score between two tokens is the dot product of one’s query with the other’s key, scaled and passed through a softmax so the scores sum to one. Each token’s output is then the weighted sum of the values. In self-attention the queries, keys and values all come from the same sequence; in a decoder the mask prevents a token from seeing anything after itself.

  • Multi-head attention: the projection is repeated in parallel with several smaller heads, each free to learn a different relationship (syntax, coreference, position), and the results are concatenated
  • Quadratic cost: every token attends to every other, so compute and naive memory scale with the square of sequence length, which is why doubling a context window costs far more than double
  • FlashAttention: Dao and colleagues showed in 2022 that computing attention in tiles that stay in fast GPU memory gives the exact same result with linear memory and much higher speed; it is now standard in inference servers
  • Grouped-query and multi-query attention: share keys and values across heads to shrink the KV cache at little quality cost

Attention is also the mechanism behind two everyday LLM behaviours. Models attend more reliably to the start and end of a long prompt than to the middle, which is why instruction placement matters, and a token can only use information that is present in the sequence, which is why retrieval and good context design improve answers more than clever phrasing.

In the Ethora ecosystem

Teams building on the Ethora AI SDK never touch attention directly, but its cost structure sets the budget for every agent. The per-agent system prompt, retrieved knowledge and the per-room transcript memory all compete for the same quadratic-priced context, so an agent that is given a focused knowledge base and a trimmed transcript answers faster and more accurately than one handed everything.

For self-hosted deployments the choice of model architecture is a hardware decision. A model with grouped-query attention and FlashAttention support, served through an OpenAI-compatible endpoint such as vLLM, fits more concurrent rooms on one GPU than an older architecture of the same size. Because each Ethora agent can use a different model, a dedicated server can run a small efficient model for routine rooms and reserve a larger one for the conversations that need it.

Get started

Build AI agents on your own stack

Ethora’s AI SDK brings agents, RAG and self-hosted LLMs into your product. Talk to our team.

Start Free
Free tier available Enterprise SLA No vendor lock-in