Ollama

General definition

Ollama is an open-source runtime (MIT licence) that packages model download, storage and inference into a single tool. You install it on macOS, Linux or Windows, run ollama pull llama3.2 to fetch a model from its library, then ollama run llama3.2 to chat with it in the terminal. Under the hood it uses the llama.cpp engine and model files in the GGUF format, so it runs on plain CPUs as well as NVIDIA, AMD and Apple Silicon GPUs.

What makes Ollama useful beyond the command line is the local server it starts automatically. It listens on port 11434 and offers two APIs: its own (/api/generate, /api/chat, /api/embed) and an OpenAI-compatible one under /v1, so any client or SDK written for the OpenAI API can be pointed at a local model by changing the base URL. The server supports streaming, tool calling, structured JSON output and embeddings, and it unloads idle models to free memory.

• Model library: Llama, Qwen, Gemma, Mistral, DeepSeek, Phi and many more, in several sizes and quantisations, plus any GGUF file from Hugging Face
• Modelfiles: a short text file (FROM, PARAMETER, SYSTEM) that bakes a base model, sampling settings and a system prompt into a named custom model
• Hardware: CPU-only works for small models; a GPU with enough memory makes 7B to 30B models comfortable; memory, not raw compute, is the usual limit
• Also available as a desktop app with a chat window, and as official Docker images for servers

Compared with its neighbours: LM Studio is a desktop app with a graphical interface aimed at the same single-user case; vLLM and similar servers are built for many concurrent users on data-centre GPUs. Ollama sits in between: ideal for developers, pilots and small internal deployments, workable for a modest production load, and the natural first step in any local LLM project because it makes the model part trivial.

In the Ethora ecosystem

Ethora’s AI service talks to any OpenAI-compatible endpoint, and Ollama is the one we document first. The official ethora-ollama-example repository shows the whole switch: four environment variables (AI_API_URL, AI_API_KEY, AI_CHAT_MODEL, AI_EMBEDDING_MODEL) point the self-hosted AI agent stack at a local Ollama server instead of a cloud API. For RAG, the knowledge base runs on Postgres with pgvector and 256-dimension embeddings, so you pull an embedding model into Ollama alongside the chat model.

Two practical notes from that example. The model endpoint is service-wide: each agent can be given its own model name, but all agents share the one endpoint. And on a 4 vCPU, 8 GB CPU-only virtual machine a 3B model answered in 6 to 8 seconds per short reply, which is fine for an internal assistant or a pilot and a signal to add a GPU before a customer-facing launch. Combined with a self-hosted Ethora server, the result is a chat and AI stack with no outbound traffic to a model provider.