What is LoRA in machine learning?

LoRA, short for Low-Rank Adaptation, is a fine-tuning technique that leaves a pre-trained model's weights frozen and trains two small matrices per adapted layer whose product represents the change. It cuts trainable parameters by orders of magnitude, fits on a single GPU, and produces a small adapter file that can be loaded on top of the base model or merged into it.

What is the difference between LoRA and QLoRA?

QLoRA applies LoRA to a base model that has first been quantised to 4 bits, with a few tricks to keep training stable. The adapter itself is trained in higher precision. The benefit is memory: QLoRA lets you fine-tune models that would not otherwise fit on one consumer or workstation GPU, at some cost in training speed and a small cost in quality.

Should I use LoRA or RAG for my chatbot?

Use RAG when the model needs your facts: product details, policies, documentation, anything that changes. Retrieval updates instantly and can cite sources. Use LoRA when the model needs a behaviour: a fixed style, a strict output format, domain vocabulary or a classification scheme, and a prompt alone cannot hold it. Many production systems use both, a lightly tuned model answering over retrieved documents. Start with a prompt and retrieval; add LoRA only if evaluation shows a gap.

How much does LoRA fine-tuning cost?

Far less than full fine-tuning. A QLoRA run on a 7-billion-parameter model typically takes hours on one modern GPU, so the compute is cheap to rent or own. The real costs are the curated training set, an evaluation set to confirm the adapter helps, and retraining whenever you upgrade the base model. Serving cost is unchanged once the adapter is merged.

Glossary

LoRA (Low-Rank Adaptation)

Fine-tune a billion-parameter model by training a few million parameters. LoRA freezes the original weights and learns small add-on matrices, so customising a language model fits on one GPU and ships as a file measured in megabytes.

General definition

LoRA (Low-Rank Adaptation) is a way to fine-tune a large neural network without touching most of it. Introduced by Microsoft researchers in 2021, it rests on an observation: the change a fine-tune makes to a weight matrix has low intrinsic rank, so instead of updating the full matrix W you learn two thin matrices A and B whose product approximates the update. W stays frozen; only A and B train. For a 7-billion-parameter model a typical LoRA trains a fraction of one percent to a few percent of the parameters, needs far less GPU memory, and produces an adapter file of tens to a few hundred megabytes rather than a copy of the whole model.

Practical properties that follow from that design:

Adapters are swappable: one base model can serve many customers or tasks by loading different adapters, and some inference servers hot-swap them per request
Merging: an adapter can be folded into the base weights to produce a normal model file with no extra inference cost
QLoRA: the same method on a base model quantised to 4 bits, which brought fine-tuning of large models to a single consumer GPU
Rank and target layers (often the attention projections) are the main knobs; higher rank means more capacity and a larger adapter

LoRA is the right tool for teaching a model a behaviour: a house style, a strict output format, a domain’s vocabulary, a classification scheme, or a smaller model imitating a larger one. It is the wrong tool for teaching it facts that change. Facts belong in retrieval (RAG), where a document update takes effect immediately and every answer can cite a source; a fine-tuned model has no such traceability and forgets or blurs specifics. And before either, a well-crafted system prompt with a few examples covers a surprising share of cases at zero training cost. The usual order is prompt, then retrieval, then LoRA, and only then full fine-tuning.

Costs are modest but not zero: a curated dataset of hundreds to thousands of examples, an evaluation set to prove the adapter helps, and a training run that for a 7B model with QLoRA takes hours on one GPU. The hidden cost is maintenance: every base-model upgrade means retraining the adapter.

Prefer to watch? LoRA explained in about two minutes.

In the Ethora ecosystem

Ethora does not train models, and the AI SDK does not need to: an agent’s persona comes from its system prompt and its facts from retrieval over a knowledge base, which is the prompt-then-RAG path the section above recommends. LoRA enters when a team wants a self-hosted model with a particular behaviour: a compliance-toned assistant, a clinical shorthand expander, a model that always emits a fixed JSON schema. They fine-tune a LoRA adapter, merge it or serve it through vLLM, Ollama or another OpenAI-compatible endpoint, and set that endpoint as the agent’s model in Ethora.

The result behaves like any other agent: it joins rooms and channels, answers in the website widget, is moderated by Trust and Safety, and its conversations are archived with an audit trail. Because model choice is per agent, a LoRA-tuned model and a general cloud model can coexist in one app, and the customised model stays inside the customer’s own infrastructure, the private LLM deployment regulated buyers usually need.

LoRA (Low-Rank Adaptation)

General definition

In the Ethora ecosystem

Build AI agents on your own stack

Talk to our team

Pick a time that works for you