Glossary
Inference (AI)
AI inference is the process of running a trained model on new input to generate a prediction, classification, or response. For language models, inference is what happens every time a user sends a message and the model produces a reply.
General definition
Inference is the runtime phase of a model’s life cycle, distinct from training (where weights are learned) and fine-tuning (where weights are updated on new data). During inference, input tokens are passed through the model’s layers in a single forward pass to produce output tokens. The model weights are fixed; no learning occurs.
For large language models, inference is computationally demanding and is the primary cost driver for teams running models at scale. Key factors include model size, context window length, quantization level, batch size, and hardware (GPU, CPU, or dedicated AI accelerators). Inference serving frameworks such as vLLM and llama.cpp optimise throughput and latency for different hardware setups.
- Latency (time to first token and total generation time) affects real-time chat user experience directly
- Throughput (tokens per second across concurrent users) determines how many requests a single server can handle
- Quantization and model distillation reduce per-inference cost at the price of some accuracy
- Batching multiple requests together improves GPU utilisation but can increase individual request latency
- Speculative decoding and caching techniques are used to accelerate inference in production systems
In the Ethora ecosystem
When you deploy a self-hosted LLM agent through the Ethora AI SDK, inference runs on your own hardware. You control the model, the serving framework, and the hardware configuration, which means latency, cost, and data exposure are all within your control rather than determined by a third-party API provider.
For chat applications that need real-time responses, inference latency is visible to end users in every reply. Teams running self-hosted models choose hardware and serving frameworks to hit acceptable latency targets without incurring per-token API costs that scale linearly with usage. In healthcare and finance, keeping inference inside the organisation also satisfies data-residency requirements that prohibit sending input data to external model APIs.
