The death of Chinchilla, in one draggable point

Fix a compute budget. Every model you could train with it lives on the curve below, trading parameters against training tokens. Chinchilla picks the bottom of the curve. Real labs do not. Drag the point and switch the objective to see why.

Parameters N
8.0 B
Train tokens D
160 B
Tokens per param
20
Predicted loss
2.05
3e22 FLOP

Loss uses the Chinchilla parametric form L(N,D) equal to E plus A over N to the alpha plus B over D to the beta, with C approximately 6 N D. Values from Hoffmann et al. 2022. Absolute numbers are illustrative, the shape is the point.