Glossary
vLLM
vLLM is an open-source serving engine that makes it practical to run large language models at high throughput on your own hardware, with a drop-in OpenAI-compatible API.
General definition
vLLM is an open-source library and inference server originally developed at UC Berkeley. It is designed to maximise the number of tokens a single GPU or GPU cluster can generate per second, using a technique called PagedAttention that manages the model’s key-value cache more efficiently than naive implementations.
- Drop-in OpenAI-compatible REST API so existing code needs minimal changes
- Support for continuous batching, which processes requests as they arrive rather than waiting to fill a fixed batch
- Compatibility with a wide range of open-weight model families including Llama, Mistral and Qwen
- Tensor-parallel and pipeline-parallel modes for multi-GPU inference
Because vLLM runs entirely on infrastructure you control, it is a popular choice for organisations in regulated industries that cannot send prompts or responses to a third-party API endpoint. It pairs naturally with a private LLM deployment strategy where data must not leave the organisation’s environment.
In the Ethora ecosystem
Ethora’s self-hosted LLM AI agent layer is designed to work alongside inference servers like vLLM. When you self-host your chat and AI stack, you can point Ethora’s AI SDK at a local vLLM endpoint instead of a cloud API, keeping every message, prompt and response inside your own network.
This matters most for teams building for healthcare, finance or other regulated verticals where data sovereignty is non-negotiable. The self-hosted chat server carries the messaging layer; vLLM (or a compatible server) carries the model layer. Both run on your infrastructure.
