The memory wall. Three exponentials pulling apart.
Across a decade of accelerator generations, peak compute has grown far faster than the bandwidth that feeds it. Start every curve at the same point and drag through the generations. Watch compute run away, and watch the accelerator starve.
Peak compute
Memory bandwidth
Interconnect bandwidth
Starvation gap
16x more compute than the pipe can feed
Two exponentials with different bases diverge without limit. Compute roughly triples every two years. The bandwidth that feeds it grows closer to one and a half times over the same window. Nothing in that arithmetic ever lets bandwidth catch up, so the natural state of a modern chip is starving. That is why the scarce resource in pretraining is bandwidth, not FLOPs, and why so many design choices are really attacks on the bytes you have to move rather than the math you have to do.
Growth rates are illustrative (compute 3x, memory bandwidth 1.5x, interconnect 1.4x per two year generation), chosen to match the shape of the memory wall reported by Gholami and colleagues. They are not measurements of any specific product.