Glossary
Text Generation Inference (TGI)
Text Generation Inference is Hugging Face’s answer to running open models in production: a Docker image that loads a model from the Hub and serves it with batching, multi-GPU support and an OpenAI-style chat API.
General definition
Text Generation Inference (TGI) is a production inference server built by Hugging Face to run transformer language models behind an HTTP API. It is written in Rust (the router and scheduler) and Python (the model shards), ships as a Docker image, and is the engine behind Hugging Face’s own hosted Inference Endpoints. It is released under the Apache 2.0 licence.
- Continuous batching, so new requests join a running batch instead of waiting for it to finish
- Tensor parallelism across several GPUs for models that do not fit on one
- Token streaming over Server-Sent Events, and guided generation with JSON schemas or grammars
- Quantised weights (GPTQ, AWQ, bitsandbytes, FP8 and others) and optimised attention kernels such as FlashAttention and PagedAttention
- Prometheus metrics and OpenTelemetry tracing built in
TGI has its own /generate and /generate_stream endpoints, and since early 2024 a Messages API at /v1/chat/completions that follows the OpenAI chat format, so OpenAI client libraries work against it unchanged. Models load straight from the Hugging Face Hub by repository name, which is the main reason teams already on the Hub choose it. Embeddings are not part of TGI; Hugging Face ships a separate Text Embeddings Inference server for those.
Against the alternatives: vLLM is the most common choice today and usually leads on raw throughput and breadth of model support; SGLang adds radix-tree prefix caching and strong structured output; TGI’s strengths are tight Hub integration, a mature Docker deployment path and a stable API. All three are forms of model serving for the same open-weight models, and all three present the same OpenAI-style interface to the application, so the choice can be changed later without touching client code.
In the Ethora ecosystem
Ethora’s AI agents are served by whichever OpenAI-compatible endpoint the AI service is configured with, and TGI’s Messages API qualifies. The switch is four environment variables (AI_API_URL, AI_API_KEY, AI_CHAT_MODEL, AI_EMBEDDING_MODEL), the same ones the official Ollama example documents; the self-hosted AI agent stack then sends chat calls to your TGI deployment instead of a cloud provider. Because TGI serves generation only, the embeddings that RAG stores in Postgres with pgvector (256 dimensions) need an embedding endpoint alongside it.
The endpoint is service-wide: every agent in the deployment shares it, while each agent keeps its own model name, system prompt and knowledge. That matches the way TGI is usually run, as one well-tuned server for one or two models, and it means an Ethora installation can move between TGI, vLLM and SGLang as a serving decision rather than an application change. For regulated deployments the point is the same as with any self-hosted server: prompts, transcripts and retrieved documents stay inside your network.