llama.cpp is an open-source inference engine written in C and C++ that runs large language models on ordinary CPUs and consumer GPUs. It loads models in the GGUF format, uses quantised weights to fit them in limited memory, and provides a command-line tool, an HTTP server with an OpenAI-compatible API, and bindings for many languages.

Does llama.cpp need a GPU?

No. llama.cpp was built to run on CPUs and does so on any x86 or ARM machine, using quantised weights to fit models in ordinary RAM. A GPU makes it much faster: it supports Apple Metal, NVIDIA CUDA, AMD ROCm, Vulkan and SYCL, and can offload only some layers to the GPU when its memory is limited.

What is the difference between llama.cpp and Ollama?

llama.cpp is the engine; Ollama is a tool built on top of it. Ollama adds a model library, one-command downloads, automatic GPU detection, model lifecycle management and its own API next to the OpenAI-compatible one. Use llama.cpp directly for fine control, new architectures, embedded targets or the smallest footprint; use Ollama for convenience.

Can llama.cpp serve an OpenAI-compatible API?

Yes. The llama-server binary exposes /v1/chat/completions, /v1/completions and /v1/embeddings in the OpenAI format, with streaming, JSON-schema and grammar-constrained output, and parallel request slots. Any OpenAI client library, and any application that reads a base URL from configuration such as an Ethora AI agent deployment, can use it as the model backend.

Glossary

llama.cpp

llama.cpp is the engine that made running a language model on a laptop normal. Written in C and C++, it loads quantised GGUF models on almost any hardware and sits underneath Ollama, LM Studio and most other local AI tools.

General definition

llama.cpp is an open-source inference engine written in C and C++, started by Georgi Gerganov in March 2023 to run Meta’s first Llama model on a MacBook. It has since become the most widely deployed engine for running language models outside the data centre. It is built on the ggml tensor library, needs no Python or CUDA for the basic CPU build, and is released under the MIT licence.

Model format: GGUF, a single-file container holding weights, tokenizer and metadata, designed for fast memory-mapped loading
Quantisation: converts 16-bit weights to 2 to 8 bit integer formats (the Q4_K_M, Q5_K_M and Q8_0 family) so a 7B model fits in roughly 4 to 8 GB of RAM
Backends: optimised CPU kernels (AVX, NEON), Apple Metal, NVIDIA CUDA, AMD ROCm, Vulkan and SYCL, with partial offload of layers to the GPU when its memory is short
Tools: llama-cli for interactive use, llama-server for an HTTP API, llama-quantize to convert weights, and a Python script that converts Hugging Face checkpoints to GGUF

llama-server is the part most applications use. It exposes OpenAI-compatible endpoints (/v1/chat/completions, /v1/embeddings) with streaming, grammar-constrained and JSON-schema output, and parallel request slots with continuous batching, so a single binary can serve a small team. Bindings exist for Python (llama-cpp-python), Node.js, Rust, Go and other languages.

Most people meet llama.cpp indirectly. Ollama wraps it with model management and a friendlier API, LM Studio wraps it in a desktop interface, and projects such as GPT4All, Jan and KoboldCpp build on it too. Using llama.cpp directly makes sense when you need a specific build flag, support for a new model architecture before the wrappers catch up, an embedded or edge deployment, or the smallest possible footprint. For many concurrent users on data-centre GPUs, vLLM and similar servers deliver more throughput; llama.cpp optimises for running anywhere rather than for peak load.

In the Ethora ecosystem

Ethora does not bundle llama.cpp, but its AI service works with any OpenAI-compatible endpoint, and llama-server provides one. Four environment variables (AI_API_URL, AI_API_KEY, AI_CHAT_MODEL, AI_EMBEDDING_MODEL) point the self-hosted AI agent stack at it, with Postgres and pgvector holding the 256-dimension embeddings for RAG. In practice most teams reach llama.cpp through Ollama, which is the setup documented in the official ethora-ollama-example repository.

The reason this engine matters to Ethora users is cost of entry. A self-hosted Ethora server plus a llama.cpp-based model on a CPU box is the cheapest complete chat-and-AI stack with no data leaving the network: in the Ollama example, a 3B model on a 4 vCPU, 8 GB virtual machine answered short replies in 6 to 8 seconds. That is enough for an internal assistant or a pilot in a healthcare or finance setting; add a GPU, or move to vLLM, when a customer-facing load arrives.

llama.cpp

General definition

In the Ethora ecosystem

Build AI agents on your own stack

Talk to our team

Pick a time that works for you