Glossary

Self-Hosted AI

Running the model, the vector store and the chat or agent application on servers you operate. You gain control over data and cost; you take on GPUs, updates and security.

General definition

Self-Hosted AI means the AI workload runs on compute you administer: an open-weight language model served by an inference engine, an embedding model and vector store for retrieval, and the application layer (chat, agents, workflows) that end users touch. The hardware can be rented in your own cloud account or sit in your building; what matters is that no vendor-operated AI service is in the request path. It is the AI counterpart of running your own database or mail server, and the same trade-offs apply.

What you gain:

  • Data control: prompts, documents and outputs stay inside your network boundary, which simplifies GDPR, HIPAA and sector rules about third-party processors
  • Cost predictability: a GPU server costs the same at 2 a.m. as at peak; there is no per-token bill that grows with usage
  • Model choice and stability: you pin a specific open-weight model version and it does not change under you
  • No egress dependency: the system keeps working if a vendor has an outage or changes its terms

What you take on:

  • Hardware: GPU memory decides which models you can run; a quantised 7B to 14B model fits on one consumer-class GPU, larger models need several data-centre GPUs, and CPU-only inference works for small models at a few seconds per reply
  • Operations: model updates, inference engine upgrades, monitoring, backups and capacity planning
  • Security: patching, access control and secrets for the model endpoint, which is now an internal service that other systems trust

The usual stack is Ollama for simple setups and vLLM for throughput, both exposing an OpenAI-compatible API, with open-weight models such as Llama, Mistral, Qwen or Gemma. Many teams start on-premise or in a private cloud VPC and keep a cloud API as a fallback for the tasks a small local model handles poorly.

In the Ethora ecosystem

Ethora’s self-hosted edition is the whole chat and AI platform as containers: the Node.js API and jobs, ejabberd for messaging, MongoDB, MySQL, Redis, Centrifugo, MinIO file storage and Caddy for TLS, installed with Docker Compose or the Helm chart on Kubernetes. Ethora Core is free to run with per-server limits, and the core on GitHub is AGPL licensed. The AI service talks to any OpenAI-compatible endpoint, so pointing it at Ollama or vLLM on the same network is a change of four environment variables (API URL, key, chat model, embedding model); RAG over your knowledge base then uses Postgres with pgvector and 256-dimension embeddings from your local embedding model.

The official Ollama example documents the whole setup, including a measured baseline: on a 4 vCPU, 8 GB CPU-only VM a 3B model answered short replies in roughly 6 to 8 seconds, fine for internal assistants and too slow for a public-facing bot. The model endpoint is configured service-wide (each agent can choose a model name, not a separate endpoint). The self-hosted LLM AI agent page covers the details; if you want the data-control benefits without the operations, a managed dedicated server run by Ethora in your cloud account is the no-ops route.

Get started

Build AI agents on your own stack

Ethora’s AI SDK brings agents, RAG and self-hosted LLMs into your product. Talk to our team.

Start Free
Free tier available Enterprise SLA No vendor lock-in