Glossary
Private LLM
A private LLM is a large language model you deploy within an environment your organisation controls, so no query, prompt or response is processed by a third-party provider.
General definition
A private LLM is any large language model deployment where inference runs on infrastructure owned or exclusively leased by your organisation. This includes local models on company servers, virtual machines in a private cloud tenancy, and air-gapped on-premises clusters. The common thread is that the model provider’s infrastructure is not involved at inference time.
- Data sovereignty: prompts, responses and any retrieved documents never leave your network
- Compliance readiness: supports HIPAA, GDPR, FCA and other regulations that restrict data processing by third parties
- Model customisation: you can fine-tune or instruct the model with proprietary data that you would not send to an external API
- Cost predictability: GPU server costs are fixed rather than per-token
Private LLM deployments typically use open-weight models such as Llama, Mistral or Qwen, served by an inference engine like vLLM. They are often combined with retrieval-augmented generation to ground the model in your organisation’s own documents without those documents leaving your network.
In the Ethora ecosystem
Ethora’s self-hosted LLM AI agent offering is designed for exactly this pattern. The AI SDK exposes a configurable model endpoint: point it at your private LLM server and Ethora’s chat and agent features, including RAG bots, routing assistants and in-thread summarisation, run entirely within your network.
This makes Ethora a natural fit for healthcare platforms, financial services and any sector where patient or customer data cannot touch an external API. The self-hosted chat server handles messaging; your private LLM handles intelligence. Both stay on your infrastructure.
