Glossary
Continuous Batching
Continuous batching is the scheduling trick that lets one GPU hold dozens of conversations at once: requests join and leave the batch token by token, so no slot sits idle waiting for the longest answer to finish.
General definition
Continuous Batching (also called iteration-level scheduling or in-flight batching) is how modern inference servers keep a GPU busy while serving many conversations at the same time. It was introduced in the Orca paper from Seoul National University at OSDI 2022 and is now standard in vLLM, Text Generation Inference, SGLang, TensorRT-LLM and llama.cpp’s server.
A language model generates one token per forward pass. With static batching, a server collects a group of requests, runs them together, and returns only when the longest has finished; short answers sit idle holding GPU memory while the long one completes, and new arrivals wait for the next batch. Because response lengths vary wildly in chat, a static batch spends much of its time partly empty.
- The scheduler runs the model one decoding step at a time across all active sequences
- When a sequence emits its end-of-sequence token it leaves the batch immediately and its memory is released
- A waiting request is admitted at the next step, as soon as there is room in the KV cache
- Prefill (processing a new prompt) is interleaved with decode steps or chunked so one long prompt cannot stall everyone else
The payoff is throughput: published comparisons show large multiples over static batching on the same hardware, because no slot is ever idle. The cost is a little per-token latency for each user, since every step now serves a bigger batch, and memory pressure, which is why continuous batching arrived together with paged KV-cache management in vLLM. For a single user on a laptop the technique changes nothing; it matters the moment a model serving endpoint has a queue.
In the Ethora ecosystem
Ethora does not implement continuous batching; the inference server behind its OpenAI-compatible endpoint does. But it is the reason an Ethora deployment can serve many rooms at once from one GPU: every agent reply, every RAG answer and every multi-agent exchange becomes a request to the service-wide model endpoint, and a server such as vLLM or SGLang interleaves them at the token level. The self-hosted AI agent stack is configured with four environment variables, so moving from a single-user runner to a batching server is a URL change.
The practical signal to watch is queueing. The official Ollama example measured a 3B model on a 4 vCPU, 8 GB CPU-only machine at 6 to 8 seconds per short reply; that is one conversation at a time. When several rooms are active, per-reply time stretches unless the endpoint batches. A GPU with a continuous-batching server is the usual next step before a customer-facing launch, and the heartbeat scheduler and multi-agent rooms, which generate requests without a human typing, are worth including when sizing it.