Scaling Laws for Neural Language Models (2020)

This paper discovered that language model performance follows predictable mathematical relationships (power laws) with model size, dataset size, and compute budget. These scaling laws enable researchers to predict how well a model will perform before training it.

The authors systematically varied model size (from thousands to billions of parameters), dataset size, and training compute, then measured the resulting performance (cross-entropy loss on text prediction). They found smooth, predictable power-law relationships across seven orders of magnitude.

Key findings: Performance depends most strongly on scale (parameters, data, compute) and very weakly on architectural details like depth vs. width. Larger models are more sample-efficient (they learn more from each training example). There are optimal trade-offs between model size and training data for a given compute budget.

Scaling laws gave AI labs a scientific basis for investment decisions. Instead of training massive models and hoping for the best, they could predict performance in advance and allocate resources optimally. This is why AI labs spent billions on training: the scaling laws predicted that the investment would pay off, and they were right.

These laws explained why GPT-3 was so much better than GPT-2 (it was not just bigger, it was on a different point of the scaling curve) and predicted that further scaling would continue to improve performance.

The Chinchilla paper (2022) later refined these laws, showing that most models were over-parameterized and under-trained. The optimal strategy is to train smaller models on more data, which directly influenced LLaMA and subsequent efficient model designs.

Authors: Jared Kaplan, Sam McCandlish, Tom Henighan, and 7 others (Johns Hopkins, OpenAI).

Key insight: Loss scales as a power law with model size N, dataset size D, and compute C.

Link to paper: https://arxiv.org/abs/2001.08361

Full paper: https://arxiv.org/abs/2001.08361

Hoffmann et al., "Chinchilla" (Training Compute-Optimal LLMs) - https://arxiv.org/abs/2203.15556

Nostalgebraist: "Scaling laws literature review" - https://www.lesswrong.com/posts/