Small Language Model (SLM)

General definition

A Small Language Model (SLM) is a generative language model built with far fewer parameters than the frontier models behind the big chat assistants. There is no fixed cut-off, but models from roughly one to ten billion parameters are usually called small, and some run at a few hundred million. They are trained with the same transformer techniques as a full-size LLM, often on carefully filtered or synthetic data and frequently by distilling a larger model, which is why recent SLMs punch well above their size on reasoning and coding benchmarks.

• Examples: the Microsoft Phi family, Google Gemma, the small Llama variants from Meta, Mistral 7B and the smaller Qwen models
• Where they run: a single consumer GPU, a laptop with unified memory, an edge device or a phone; on-device assistants in phones and browsers are SLMs
• How they are served: through local runtimes such as Ollama and llama.cpp, or an inference server such as vLLM exposing an OpenAI-compatible API
• How they are shrunk further: quantization to 8-bit or 4-bit weights cuts memory use by two to four times with a modest quality cost

The case for an SLM is rarely raw capability. It is latency (no network round trip), cost per token (a fixed box instead of metered API calls), privacy (prompts and documents never leave the machine) and predictability (the weights do not change under you). The case against is breadth: a small model knows less, follows complex instructions less reliably and needs a good retrieval layer to stay accurate on facts. In practice SLMs shine on narrow, well-defined jobs: classification, extraction, summarising a known document set, answering questions over a curated knowledge base.

For regulated organisations the privacy argument often decides it. A private LLM or local LLM is usually an SLM or a mid-size open-weight model, self-hosted so that patient records, financial data or customer conversations are never sent to a third-party API.

In the Ethora ecosystem

Ethora treats the model as a per-agent setting. Each agent in the AI SDK can point at a cloud model API or at a self-hosted model through an OpenAI-compatible endpoint, which is exactly how Ollama and vLLM serve small models. A support agent can run on a quantized 7B or 8B model on one GPU while a drafting agent in the same app uses a larger cloud model, and the choice can change without touching application code.

On a dedicated or self-hosted deployment the SLM, the retrieval index and the chat transcripts all sit in the customer’s own cloud account or on-premise, the pattern described on the self-hosted LLM agent page. RAG over the agent’s knowledge base compensates for the smaller model’s narrower knowledge, and the same Trust & Safety, audit trail and retention controls apply regardless of which model produced the reply.