What is a Tensor Processing Unit (TPU)?

A tensor processing unit, or TPU, is a chip designed by Google to do one kind of arithmetic: multiplying grids of numbers. That is the work neural networks spend most of their time on. A tensor is just such a grid, so the chip is named after the thing it multiplies. Chips sit on a ladder, and each rung trades away more flexibility for more speed at one task. A CPU keeps a few cores clever enough for anything; a GPU keeps thousands of simple cores for parallel work; a TPU casts the one operation into its wiring.

A tensor processing unit is a chip that does one kind of arithmetic, the grid multiplication neural networks are built on, and almost nothing else. Google designed it that way. Every ability a chip carries costs room and power even while it sits unused: circuitry for deciding what runs next, for handling any kind of program. Strip those away and the freed room holds more multiply units. The first TPU, in Google's data centers from 2015, shows what this buys. One multiply unit does a tiny job: it multiplies a pair of numbers. That chip carried 65,536 of them working at once and swept a whole grid in one pass, where an ordinary processor takes a few numbers at a time. The design has a cost. A TPU cannot be pointed far from its one operation, so it earns its keep where that operation runs at enormous volume. Google, running neural networks all day, is such a place, which is why a search company designs chips. Most other AI work still runs on more flexible chips, partly because TPUs are mostly rented through Google's cloud, and partly because flexibility is safer while models keep changing. Early generations could only run finished models, taking questions in and producing answers; later ones do the training as well. What has never changed is the trade: the abilities left out of the wiring cannot be added back.

A tensor processing unit is a printing press, where an ordinary processor is a hand that writes. The hand can produce any page you ask of it, at a hand's pace. The press produces exactly one page, the one set into its type, and setting that type is slow and costly. But once the type is set, copies pour off the press by the thousand, faster than any number of hands. Change the page and the press must be re-set; ask for a poem mid-run and it cannot oblige. Presses never replaced pens. They took over the one situation that repays the setup: the same page, wanted in enormous numbers. That is the TPU: one page, set in silicon, printed all day.