LLM Quantization: How It Works and Memory Savings

LLM quantization is the practice of storing language model weights in lower-precision numbers, such as 8-bit or 4-bit instead of 16-bit, so the model uses less VRAM and can run on cheaper hardware. When moving from 16-bit to 4-bit, weights take about 4x less space: an 8-billion-parameter model uses roughly 4 GB instead of 16 GB. The trade-off is possible quality loss, which grows with heavier compression and depends on the method and the task.

Prepared by AIrassvet October 21, 2026 for executives and IT directors planning to run open language models on their own servers and wanting to understand what drives hardware costs. The explanation is based on research papers on LLM.int8(), GPTQ, AWQ, SmoothQuant, QLoRA, and FP8, llama.cpp and vLLM documentation, and model cards for Qwen and Gemma. We did not conduct our own quality measurements for this article: all measurements belong to the source authors, and calculations are marked.

At a glance
What’s included and not included in this article
What quantization is in simple terms
How model quantization works
How much memory quantization saves
Main methods and formats
Where quantization is used in business
Limitations and risks
How to choose a quantization level
Glossary
FAQ
Primary sources and methodology

At a glance

Quantization reduces memory for weights in proportion to the number of bits: 16 → 8 bits — by half, 16 → 4 bits — by about four times.
8-bit quantization usually has almost no loss. The authors of LLM.int8() (2022) showed inference on models up to 175 billion parameters in 8 bits without quality degradation.
4-bit is a practical compromise. GPTQ (2022) compresses weights to 3–4 bits with, according to the authors, a negligible loss of accuracy.
2–3 bits is the risk zone. A 2024 study on LLaMA3 quantization reports noticeable degradation at ultra-low bit widths.
You need to test on your own tasks: average benchmarks do not guarantee quality on your company’s Russian-language documents.

What’s included and not included in this article

We explain quantization of weights and cache in text-based language models for inference — that is, for running an already trained model. We do not cover quantization during training from scratch, pruning, or distillation as separate compression methods, nor do we provide step-by-step setup for specific servers. Model and hardware selection is covered in full in the 2026 local LLM overview, and the comparison of cloud vs. in-house server costs is covered in the article “Cloud or local LLM: TCO”.

What quantization is in simple terms

A language model is billions of weight numbers. Usually each is stored in a 16-bit format (FP16 or BF16), meaning it takes 2 bytes. Quantization replaces these numbers with approximations from a small set of levels: 256 levels in 8 bits or 16 levels in 4 bits. It is similar to saving a photo with fewer colors: the image is still recognizable, the file is smaller, but subtle shades are lost.

The model is not retrained from scratch. In most methods, quantization is performed after training (post-training quantization, PTQ): ready-made weights are recalculated in minutes or hours. For example, the authors of GPTQ report quantizing a 175-billion-parameter model in about 4 GPU-hours. There is also an approach where the model is further trained with future rounding in mind — quantization-aware training (QAT); Google, for example, uses it for Gemma 4 versions.

How model quantization works

The simplest approach is to find a scale for a group of weights and round each value to the nearest level. In practice, it is more complicated because large language models contain outliers: a small number of very large values that have a strong impact on quality. That is why methods differ in how they protect important values:

LLM.int8() moves outlier features into a separate 16-bit multiplication, while multiplying more than 99.9% of values in 8 bits (Dettmers et al., 2022).
GPTQ rounds weights sequentially and compensates for the error with neighboring weights using approximate second-order information (Frantar et al., 2022).
AWQ identifies important channels based on activations rather than the weights themselves: according to the authors, protecting just 1% of significant weights greatly reduces quantization error (Lin et al., 2023).
SmoothQuant shifts the quantization challenge from activations to weights through a mathematically equivalent transformation and makes it possible to keep both weights and activations in 8 bits (Xiao et al., 2022).

Weights are split into small groups with their own scale, so the effective bit width is slightly higher than the nominal one. In llama.cpp tables, the Q4_K_M format for Llama 3.1 8B is 4.89 bits per weight, while Q8_0 is 8.50 bits.

How much memory quantization saves

Memory for weights is calculated simply: number of parameters × bits per weight / 8 = bytes. The calculation below was made by AIrassvet using this formula and does not account for context cache (KV-cache), activations, or framework overhead — in practice, you need to add a buffer to the weights.

Model	16 bits (FP16/BF16)	8 bits	4 bits
8 billion parameters	16 GB	8 GB	4 GB
32 billion parameters	64 GB	32 GB	16 GB
70 billion parameters	140 GB	70 GB	35 GB

Real files are slightly larger because of group scales. The llama.cpp developers list the following sizes for Llama 3.1 8B: 14.96 GiB in F16 (about 16 GB), 7.95 GiB in Q8_0, and 4.58 GiB in Q4_K_M. For Llama 3.1 70B — 280.9 GB for the original file and 43.1 GB after Q4_K_M; judging by the size — about 4 bytes per parameter — the original file in this table is 32-bit.

Quantization can also speed up text generation: it is often limited by how fast weights can be read from memory, and smaller weights are read faster. In the same llama.cpp tables, Llama 3.1 8B generates 29.17 tokens per second in F16, 50.93 in Q8_0, and 71.93 in Q4_K_M. These are measurements from the llama.cpp developers on their own hardware, not a guarantee for any server.

Main methods and formats

Method or format	Bit width	What Is Quantized	Where It’s Used
LLM.int8()	8-bit	weights, outliers in 16 bits	bitsandbytes library
GPTQ	3–4 bits	weights	pretrained models on Hugging Face, vLLM
AWQ	4 bits	weights	Qwen3-8B-AWQ from the Qwen team, vLLM
SmoothQuant	8 bits	weights and activations (W8A8)	server-side inference
FP8	8 bits	weights, activations, KV cache	Qwen3-8B-FP8, vLLM
GGUF (k-quants)	2–8 bits	weights	llama.cpp, local CPU and GPU inference
NF4 (QLoRA)	4 bits	weights during fine-tuning	fine-tuning on a single GPU
QAT	usually 4 bits	weights, training with rounding awareness	Gemma 4 QAT Q4_0

FP8 — an 8-bit floating-point format; Micikevicius and coauthors (2022) described two encodings: E4M3 (4 exponent bits and 3 mantissa bits) and E5M2 (5 and 2 bits). The vLLM documentation describes support for AWQ, GPTQ, GGUF, bitsandbytes, and FP8, as well as FP8 quantization of the context cache. NF4 — a 4-bit format from the QLoRA paper (Dettmers et al., 2023): it made it possible to fine-tune a 65-billion-parameter model on a single GPU with 48 GB of memory.

Where Quantization Is Used in Business

Your Own Model on a Single Server. Quantization moves a 70-billion-parameter model from the “multiple GPUs” class into the one- or two-card class, if there is enough memory for the weights and context cache.
Cheaper Cloud Inference. Less memory means you can rent a less expensive GPU or serve more concurrent requests on the same one. We covered how open-model inference works in the cloud in an article about open-source LLM inference in a Russian cloud.
Local Deployment on Workstations for pilots and secure environments where data cannot be sent outside: GGUF formats are designed in part for CPU use.
Fine-Tuning on Limited Hardware via QLoRA: the base model weights are stored in 4 bits, and only the adapters are trained.
Long Context. Quantizing the context cache reduces the memory consumed by long documents and many parallel sessions.

Limitations and Risks

Quality Drops Nonlinearly. A study of LLaMA3 quantization (Huang et al., 2024) evaluated 10 methods at 1–8 bits and recorded a noticeable degradation, especially at ultra-low bit widths.
Average Benchmark Results Hide Weak Spots. Losses can be unevenly distributed across tasks, and publications on quantization methods mostly measure quality on English-language datasets. That is why long-form reasoning, code, Russian, and niche terminology should be tested on your own tasks.
Not All Hardware Speeds Up All Formats. The speed benefit depends on whether the GPU and framework support a specific format; otherwise, memory is saved but time is not.
A Quantized Copy Is Not the Original. Third-party versions may differ in settings and calibration data; official versions from the model developer are preferable, and the license is inherited from the original model.
Memory Savings Do Not Equal Budget Savings. Total cost is determined by workload, latency requirements, and operations, not just VRAM usage.

How to Choose a Quantization Level

We recommend starting from the task and the validation set, not from the smallest file size:

Start with 8 bits (FP8 or INT8) if memory allows: according to LLM.int8() and SmoothQuant, losses at this level are small.
Move to 4 bits (AWQ, GPTQ, GGUF Q4_K_M, or QAT versions) when 8 bits do not fit or you need a larger model on the same hardware. A larger model in 4 bits may be more useful than a smaller one in 16 bits at the same memory footprint, but this needs to be confirmed by testing.
Do not use 2–3 bits in production without separate quality validation on your own tasks.
Compare options on one set of real requests: the original model, 8 bits, and 4 bits. We described how to build such a set in the article “Your Own Benchmark for Choosing an LLM”.

We choose the “8 bits first, then 4” order because it limits risk: each next compression is checked against an already working version, not against expectations.

Glossary

Weight (parameter) — a number inside the model that determines its behavior; models have billions of them.
FP16 / BF16 — 16-bit floating-point formats, standard for storing LLMs.
INT8 / INT4 — integer formats with 8 and 4 bits.
FP8 — an 8-bit floating-point format.
PTQ — post-training quantization, without retraining the model.
QAT — quantization-aware training, so the model gets used to rounding.
Calibration data — a small set of texts the method uses to estimate the value distribution.
KV-cache — a cache of intermediate attention values that grows with context length.
GGUF — the model file format used by the llama.cpp project.
Perplexity — a metric of how well a model predicts text; llama.cpp uses it and KL divergence to estimate quantization loss.

Frequently asked questions

Are quantization and model compression the same thing?

Quantization is one method of compression, alongside pruning and distillation. It does not remove weights or train a new model; it stores the same weights at lower precision. As a result, the architecture and capabilities of the model remain the same, while memory usage, sometimes speed, and answer quality change to varying degrees.

How much video memory does a model with 70 billion parameters need?

For 4-bit weights, about 35 GB using the formula “parameters × bits / 8”; for 8-bit, about 70 GB; and for 16-bit, about 140 GB. You also need to add memory for the context cache and overhead, which depend on prompt length and the number of concurrent users. That is why the final requirement is determined by load testing.

Can you quantize a model yourself?

Yes. For popular formats, there are open-source tools: llama.cpp for GGUF, libraries for AWQ and GPTQ, and LLM Compressor for vLLM formats. But for many models, developers already publish official quantized versions, such as Qwen3-8B-AWQ and Qwen3-8B-FP8, or QAT versions of Gemma 4. It makes sense to start with those and test your own quantization on calibration data that is close to your tasks.

Will the model perform worse in Russian?

It may, and this should be checked separately. Publications on quantization methods mostly evaluate quality on English-language benchmarks, so their conclusions do not guarantee the same result for Russian-language documents and company terminology. Compare the original and quantized models on a set of real Russian queries before deployment.

Primary sources and method

We checked the sources on October 21, 2026. The quality and speed figures belong to the authors and were obtained under their conditions; the memory table is a calculation based on the formula, without accounting for context cache.

How AIrassvet helps deploy a quantized model on your own servers

Quantization is a trade-off between hardware and quality, and it should be decided based on measurements. On the AIrassvet website you can find the service areas that address this task:

LLM implementation, including deployment of the model on the client’s servers: selecting the model and quantization level for the available hardware;
process audit and data preparation: collecting a set of real queries used to compare the original and quantized versions;
testing and launch: checking quality and load before moving the quantized model into production.

The first step is to choose one process that needs its own model and document the current result, data sources, hardware constraints, and acceptance criteria.

Discuss the project

Bottom line: quantization saves memory, and quality should be verified with testing

LLM quantization is the main way to fit a large model into available video memory: 8-bit cuts memory for weights in half, and 4-bit reduces it by about four times. Methods such as GPTQ, AWQ, and QAT keep losses at 4-bit relatively small, but with stronger compression the quality drops noticeably.

That is why you should choose the quantization level based on the task: start with 8-bit, switch to 4-bit if memory is tight, and compare the options on your own queries. That way, hardware savings do not turn into an unnoticed loss in answer quality.