What is High-Bandwidth Memory (HBM)?

High-bandwidth memory, or HBM, is computer memory built as a stack of chips placed on the same slab as the processor, joined to it by more than a thousand wires that all carry data at once. For comparison, an ordinary memory channel runs on 64 wires. The width exists because a modern processor can finish its arithmetic before a narrow connection delivers the next batch of numbers, so delivery, not arithmetic, often sets the pace. HBM widens that delivery, and being built onto the same slab keeps every trip short too. It changes how quickly numbers reach the chip, not what the chip can do with them.

High-bandwidth memory is memory redesigned around one problem: processors got faster at arithmetic than ordinary memory got at supplying the numbers they work on. That gap has a name: the memory wall, and for much of the work an AI chip does, the wall, not arithmetic speed, is the limit. HBM answers the wall in two ways. First, width. Pushing a narrow connection to run ever faster costs power and gets harder with each step, so HBM runs more than a thousand wires side by side instead. The gain comes from the count of wires, not the pace of any one. Second, nearness. The memory chips are stacked into a tower and placed on the same slab of silicon as the processor, so every trip the data makes is short and cheap. Here is what that combination gets you. In the simplest case, a model whose numbers fill 100 gigabytes is read in full for every word it writes. Memory that moves 2 terabytes each second is moving 2,000 gigabytes, so it completes that 100-gigabyte read 20 times a second. That is the ceiling: 20 words a second, and faster arithmetic cannot raise it. Only wider, closer memory can. That is why an AI chip's memory figures sit beside its speed figures on every specification, and why this memory cannot be added later: it is built into the processor's package, not plugged into a slot.

High-bandwidth memory is a wider packing bench, set right beside the shelves. Two warehouse workers pack orders with the same quick hands. One has a narrow bench that holds a handful of items, a storeroom down the corridor, and a shift that is mostly walking: pack a little, walk, restock, walk back. The other has a bench three times as wide, stacked with everything her next orders need, an arm's reach from the racks. She ships far more parcels per hour without packing any faster, because packing was never the slow part. Fetching was. That is HBM: the same worker, a wider surface, a shorter walk.