One synchronous job, 8 GPUs a node. Any failure anywhere stops all of it.
Meta’s reliability study assumes 5 minutes. Asynchronous checkpointing gets it under one.
Detect, drain the bad node, reschedule and reload the last checkpoint. The same study puts it at 5 to 20 minutes.
Checkpoint interval
Repairs and cost
Your assumption, not a published figure. Sets how many spare nodes keep the job at full size.
What if
- Goodput
- %
- Something breaks every
- Checkpoint every
- GPU-hours lost a day
- Spare nodes to hold
- nodes
- Training kept
- Checkpoint write
- Rolled back
- Restarting
- Expected
- This day
Goodput against job size · press or drag to resize the job
- The other two fleets
- Meta’s reported Llama 3 figure
What stops it, at your size
| Rank | Cause | Share | At your size | Job time lost a day |
|---|
Under the hood
A cluster is sold by the GPU-hour. A model is trained by the GPU-hours that made it into a checkpoint. The difference is published, and it never appears on the quote.
Failures scale with the job
In a synchronous job every GPU is a single point of failure. A part that fails once in six years on one GPU stops a 16,384-GPU job every three hours, and a 100,000-GPU job every half hour.
Every failure is paid twice
Once in the restart, while the bad node is drained and the job reloads, and again in the work since the last checkpoint, which is thrown away and done over. Neither shows up as an outage on any dashboard.
The checkpoint interval is the dial
Checkpoint often and the job stalls writing. Checkpoint rarely and each failure throws away more. The interval that loses least has been known since 1974: the square root of twice the stall times the time between failures.
- Between interruptions, Llama 3 on 16,384 H100s
- Of those interruptions traced to the GPU itself
- Between failures for any one GPU on that run
One rate, one square root, one ratio. The example follows whatever job you have set up.
How often the job stops
A published rate of interruptions per thousand node-days, times the nodes in your job. The rates come from whole fleets over months, so they already include the bad batches and the bad weeks.
How often to checkpoint
Young’s first-order optimum: √(2 × stall × time between failures). It balances the time spent writing against the work lost when a failure lands mid-interval, which is half an interval on average.
What each failure costs
The restart, plus half an interval of work rolled back. Both stop the whole job, so they are paid on every GPU at once.
What is left
The time between failures minus what failures cost, divided by the checkpoint stall. That is goodput, the share of paid GPU-hours that reaches a checkpoint. Meta calls it effective training time.
Three published failure rates and one published breakdown of causes. Fleets that never published a rate per node are not here, however large.
The fleets
| Fleet | Hardware | Per thousand node-days | At 16,384 GPUs | Evidence |
|---|
The causes, Llama 3 Table 5
| Cause | Category | Interruptions | Share | Note |
|---|
Every constant, and where it came from. The model is six lines.
- Failure rate
- λ = GPUs ÷ 8 × rate per node-day
- Between failures
- M = 1 ÷ λ
- Interval
- τ = √(2 × stall × M) (Young, 1974)
- Goodput
- (1 − λ × (restart + τ ÷ 2)) ÷ (1 + stall ÷ τ)
- Lost a day
- GPUs × 24 h × (1 − goodput)
- Spares
- 99th percentile of Poisson(λ × node share × repair days)
It assumes one synchronous job across the whole cluster, which is how frontier pretraining runs. Many small jobs on the same cluster lose far less, because a failure stops only the job it lands in.
Failures arrive at a constant rate. No burn-in, no wear-out, no bad weeks; the published rates average all of those in. The Llama 3 rate counts only the 419 unexpected interruptions, not the 47 planned ones.
It is first order. The goodput formula is the one Meta’s reliability study uses, and it holds while restart plus half an interval is small against the time between failures. It ignores failures during a restart, so below about half goodput it flatters the job.
The cause mix is Llama 3’s, applied to every fleet. It is the only per-component breakdown published at this scale. The paper prints 30.1 percent for faulty GPUs; 148 of 419 is 35.3, and the page uses the counts.
Spares count only the causes that take a node out: GPU, host and unplanned host maintenance, 77 percent of the total. Software, network fabric and unknown causes restart the job without losing hardware. The repair turnaround is your number, not a published one.
These are A100 and H100 fleets. Rack-scale systems like GB200 NVL72 change the failure domain from a node to a rack, and no fleet has published a rate for them at scale yet.
Part of Infrabench
Free tools for the people who build data centers, on public data with every constant on the page.
Built by Adrian Mucha
A product designer for the software most designers avoid: data center tools, AI tooling and operator consoles where a wrong click costs real money.
- See the portfolioadrianmucha.us
- Open to new workBased in Los Angeles