Glossary
Quantization
Quantization reduces the numerical precision of a model’s weights, for example from 32-bit floating point down to 8-bit or 4-bit integers, which cuts memory usage and speeds up inference with minimal accuracy loss.
General definition
Quantization is a model compression technique that replaces high-precision floating-point weights with lower-precision representations. A model trained at 32-bit or 16-bit precision can be quantized to 8-bit (Q8), 4-bit (Q4), or even lower, often with only a small drop in benchmark scores.
The practical benefit is significant: a 7-billion-parameter model that requires roughly 14 GB of VRAM at 16-bit precision can fit into about 4 GB at 4-bit, bringing it within reach of consumer and single-server GPU setups. Formats such as GGUF (used by llama.cpp) and GPTQ are common in the self-hosted LLM ecosystem.
- Post-training quantization (PTQ) applies compression after training without retraining the model
- Quantization-aware training (QAT) simulates low-precision arithmetic during training for higher accuracy at the target precision
- Mixed-precision quantization keeps sensitive layers at higher precision while compressing the rest
- Accuracy impact varies by model size and quantization level; larger models tolerate more compression
In the Ethora ecosystem
Quantization is directly relevant to teams running self-hosted LLM agents on their own infrastructure. By selecting a quantized model through the Ethora AI SDK’s configurable model endpoint, you can serve a capable model on a single GPU server rather than a multi-GPU cluster, reducing both hardware cost and the surface area exposed to compliance auditors.
For healthcare and finance deployments where data must stay on-premises, quantization often determines whether self-hosting is economically practical. A Q4 or Q8 model that fits on one machine can replace a cloud-hosted model call without sacrificing the data-residency guarantees your compliance team requires.
