What is Quantization?

Quantization shrinks an AI model by storing its numbers with less precision. Each number takes less memory, so the model runs on smaller, cheaper hardware. Done well, quality barely drops.

A model is billions of numbers, called weights, normally stored at 16 bits each. Quantization re-encodes them at 8 or 4 bits, so a 70-billion-weight model drops from about 140 gigabytes to roughly 35 at 4 bits. The full version doesn't fit even on a data-center GPU, while the shrunken one squeezes onto a single high-end consumer graphics card. The trick works because weights cluster in narrow ranges, so a well-chosen coarse scale loses little: benchmark scores typically dip a few percent at 8 bits and stay usable at 4. Below that it gets rough, and 2-bit versions are often noticeably worse. Speed usually improves as well: the slow part of running a model is fetching all those numbers from memory, and smaller numbers mean less to fetch. This is the main reason local AI exists at all, letting people run yesterday's frontier models on gaming hardware.

Shrinking a photo to send it. The original file is huge; saving it at a lower quality makes the file a tenth of the size, and on a phone screen nobody can tell. Zoom in far enough and you will find the smudges, but for everyday viewing the picture does its job.