Clicked Gallery

What is a Compute Cluster?

Highlighted from a real engineering doc. Explained by Clicked.

Used in a sentence

Engineering Notes · AI Systems

Nvidia's Jensen Huang called xAI's assembly of Colossus, a compute cluster of 100,000 chips, a “superhuman” effort, saying a machine of that size would normally take three years to plan and another year to get working.

The reader highlighted one word in the docs. Clicked explained the technical term “compute cluster” in plain language:

Explained in three depths

Same facts, different vibe — Slang mode 😎

The Clicked way

●○○

Overview

A compute cluster is a large group of separate computers, joined by fast links so that one job can run across all of them at once. Each computer is a complete machine with its own processing chips and its own memory. Because that memory belongs to it alone, anything one machine calculates must travel across the links before the rest can use it. So the job halts after each step while the machines exchange what they found, and nothing continues until the slowest reports. Size is quoted in chips rather than machines, since each machine holds several: by 2026 the largest AI training clusters were being announced at 100,000 chips and beyond.
●○○

Overview

A compute cluster is what you get when the work is far too big for one computer, so thousands of them are wired together to share it. Each one handles its own slice, and none of them knows what the others have got unless they all pause and swap notes. So the whole thing runs in rounds: everyone works, everyone pauses, everyone swaps answers, and only then does the next round start. Which means a headline figure of 50,000 processors tells you how much could be happening at any moment, and nothing about how long they spend waiting on one another. 😎

A quick take — often all you need.

●●○

Detail

A compute cluster is what you build when a job outgrows any one machine: hundreds or thousands of computers wired together to work on a single problem. Splitting a job across machines should divide the time in the same proportion, so ten machines finish in a tenth of the time. That is not what happens. Every machine must be loaded with the program and its data before any can start, and that setup takes the same time however many machines join. Say a job takes 100 days alone, with 5 of those days spent on setup. Ten machines cut the remaining 95 days to 9.5, so the job finishes in 14.5 days, not 10. A hundred machines cut those 95 days to under one, and it still takes about 6, because the 5 setup days never move. That floor has a name: Amdahl's law. Setup is only a one-time cost, though. Once the job is running, the machines have to compare what they found, typically at the end of every step, and the next step waits for the slowest to report in. Each exchange crosses the wiring, which is why the links between machines are designed and priced as carefully as the machines. A cluster whose jobs never compare results, such as a thousand unrelated simulations, manages on ordinary networking instead. Machines also fail, and each holds a piece of the task, so a breakdown halts the run rather than slowing it. Across a few thousand machines over a month that is routine, so lengthy tasks save their progress at intervals and resume from the most recent save.
●●○

Detail

A compute cluster is a few thousand computers sharing one job, and the expensive part is getting them to act as one. The job runs in rounds. Every computer does its slice, then all of them pause so each can see what the rest came up with, and nobody moves on until the last one checks in. If a single computer is a minute behind, the others sit idle for that minute, and that repeats round after round all day. Then there is the second problem: computers break down. Suppose one can be trusted to run about a year before something in it quits. A year is 8,760 hours, so with 1,000 of them going, expect one to quit roughly every 9 hours. So a training run needing three weeks will not get three clean weeks. When a computer quits, the slice of work it was holding is gone, so the run stops rather than carrying on with a hole in it. That is why the job writes down where it got to every few minutes, then rewinds to that point and carries on. Buying more of them does not solve it either, since each addition is one more thing that can quit and one more answer to wait on. Keeping a few thousand computers in step for three weeks while they quit one by one is the actual job. 😎

Want more? One click digs deeper.

●●●

Analogy

A compute cluster is a large crew putting up a tower one storey at a time. The job is too big for one bricklayer, so forty of them spread across the storey and each takes a section. Nobody can begin the storey above until every section below is finished, so the crew moves at the pace of the slowest section. The sections must also be checked against each other before the next storey starts, which takes time of its own. Add more bricklayers and each section shrinks, but the checking and the waiting grow. A cluster works the same way: each machine takes a section of every step, none moves on until all have compared results, and the more machines there are, the more of each step goes on comparing rather than working.
Deepest level reached.
●●●

Analogy

A compute cluster is a rowing boat with eight rowers in it. Each rower is strong alone, but the boat only moves well when all eight pull at the same moment, so they row in unison with one person calling the rhythm. If one falls half a beat behind, the boat does not go at seven-eighths speed; it drags, because the other seven are pulling against that one. A cluster is that boat with a thousand rowers: every machine pulls its stroke, all of them wait for the slowest, and the wiring between them calls the rhythm. 😎
Deepest level reached.

Unfamiliar concept? A real-world example makes it click — fresh analogies on tap.

AI explanations may contain errors · Not professional advice

Formal definition — The same term, explained the usual way

A compute cluster is a set of independent computers, called nodes, connected by a dedicated high-speed interconnect and managed as a single system, so that one workload can be distributed across all of them. Each node runs its own operating system and holds its own memory, and coordination is achieved by message passing over the interconnect rather than by shared memory, which is what distinguishes a cluster from a single large multiprocessor machine. Scheduling software such as Slurm or Kubernetes allocates nodes to jobs and enforces queuing policy. Clusters are the standard form of high-performance computing infrastructure, used for scientific simulation, financial modelling and, at the largest scale, the training of machine-learning models, where interconnect bandwidth and fault tolerance are usually the limiting design factors.

Want Clicked to explain terms like “compute cluster” directly in your browser — including on PDFs?

Add to Chrome — Free

50 free Explanations · No credit card required