Glossary
LLM Gateway
One endpoint in front of every model you use. An LLM gateway routes requests to providers, holds the keys, enforces limits and budgets, caches, logs and fails over, so applications never talk to a vendor directly.
General definition
An LLM Gateway is the model-serving counterpart of an API gateway: a proxy that applications call instead of calling OpenAI, Anthropic, Google, a cloud-hosted model or a self-hosted server directly. The application sees one stable interface, usually the OpenAI-compatible chat completions format, and the gateway decides which provider and model actually serve the request. That separation is what lets a team change models, add a fallback or move inference on-premise without touching application code.
- Routing and fallbacks: send requests to a model by name, policy or cost, and retry on another provider when one times out or errors
- Credentials: provider API keys live in the gateway; applications get scoped virtual keys that can be rotated or revoked
- Rate limits and budgets: per application, team or end user, so one feature cannot exhaust a shared quota
- Caching: identical or semantically similar requests are served from cache to cut cost and latency
- Logging and cost accounting: token counts, spend and traces exported to your observability stack
- Content controls: PII redaction before a prompt leaves your network and guardrail checks on responses
Gateways come as open-source proxies (LiteLLM, Portkey and Kong AI Gateway are examples), as managed services from cloud and API-management vendors, and as features built into model-serving stacks. The OpenAI-compatible API has become the common contract on both sides: most gateways speak it to applications, and self-hosted servers such as vLLM and Ollama expose it, so one gateway can front cloud and local models with the same client code.
The trade-offs are the usual ones for a proxy: an extra network hop, a component that must itself be highly available, and a concentration of sensitive traffic that needs strict access control and log hygiene. For regulated workloads the gateway is often the enforcement point for data residency, keeping prompts inside a region or inside a private LLM deployment rather than sending them to a public API.
In the Ethora ecosystem
Ethora agents are not tied to a specific model vendor. Each agent in the AI SDK has its own model setting that points at any OpenAI-compatible endpoint: a cloud model API, a self-hosted server such as Ollama or vLLM, or an LLM gateway placed in front of any of them. Pointing agents at a gateway gives you provider fallbacks, central key management, per-app budgets and request logging without changing anything inside Ethora.
This composes with Ethora’s own controls. The Trust and Safety system screens messages in rooms, the gateway screens prompts and completions at the model boundary, and the audit trail records the conversation itself. On a dedicated or self-hosted deployment the gateway and the models can run inside your own network, so inference traffic never leaves your perimeter and data residency is decided by where you deploy.
