Glossary
Model Serving
Model serving is everything between a trained model file and the application that calls it: the server, the hardware, the batching, the scaling and the API. For language models that API is now almost always the OpenAI format.
General definition
Model Serving turns a set of trained weights into a reliable endpoint. Training produces a model file; serving loads it onto the right hardware, accepts requests, batches them, runs inference, streams the output back and scales with demand. For language models the term usually means a dedicated inference server rather than a model loaded inside the application process, because a model that needs a GPU and tens of gigabytes of memory has to be shared.
- Inference server: vLLM, Text Generation Inference, SGLang, llama.cpp’s llama-server, NVIDIA Triton or Ollama; each loads the weights and runs the forward pass
- Continuous batching and KV-cache management so one GPU serves many concurrent conversations
- Hardware allocation: which GPUs, how many, tensor parallelism across them, and quantisation to fit the model in memory
- Autoscaling, health checks, request queues and rate limits, usually on Kubernetes or a managed GPU platform
- Observability: tokens per second, time to first token, queue depth, GPU utilisation and cost per request
The interface has converged. Almost every serving stack now exposes the OpenAI chat completions and embeddings format, so the OpenAI-compatible API has become the de facto contract between applications and models. That is why an application can move from a cloud model to a self-hosted one, or from vLLM to SGLang, by changing a base URL and a model name. An LLM gateway in front of several such endpoints adds routing, keys and usage accounting.
The two numbers that drive serving decisions are throughput (tokens per second across all users) and latency (time to first token and per-token speed for one user). Batching trades a little latency for a lot of throughput; a single-user desktop tool such as Ollama optimises the other way. Choosing a serving stack is mostly choosing where you want to sit on that curve, and on whose hardware the model runs.
In the Ethora ecosystem
Ethora sits on the application side of the serving boundary. Its AI agents consume a model through an OpenAI-compatible endpoint, configured service-wide with four environment variables (AI_API_URL, AI_API_KEY, AI_CHAT_MODEL, AI_EMBEDDING_MODEL); each agent then picks a model name available at that endpoint. Behind the URL you can run a cloud provider, a managed GPU platform, vLLM or SGLang on your own cluster, or Ollama on a single box, and change your mind later without touching agents, prompts or the pgvector knowledge base.
For teams that want chat and AI without running any of it, a managed dedicated Ethora server in your own cloud account covers the messaging and agent layer; the model endpoint remains yours to choose. For a fully private stack, the self-hosted Ethora Core install plus a local serving endpoint gives chat, agents and RAG with no outbound model traffic. The official Ollama example measured a 3B model on a 4 vCPU, 8 GB CPU-only machine at 6 to 8 seconds per short reply, a useful baseline when planning GPUs for the self-hosted AI agent stack.
