agent37.club 1 / 20 agent37.club

Scaling Laws &
The Bitter Lesson

Why compute always wins — and the new axes of scaling

agent37.club

The Bitter Lesson

Rich Sutton, March 2019

“The biggest lesson from 70 years of AI research is that general methods leveraging computation are ultimately the most effective, and by a large margin.”

Why it’s “bitter”

  • Researchers want their domain expertise to matter
  • Hand-engineering features is intellectually satisfying
  • It works in the short term — giving a false sense of progress
  • Then someone scales a simpler method and blows past it

The two methods that scale

  • Search — exploring more possibilities
  • Learning — extracting patterns from more data

Both leverage compute. Both beat hand-crafted knowledge. Every. Single. Time.

The Pattern Repeats

Domain Hand-Engineered Approach What Actually Won
Chess (1997) Grandmaster knowledge, clever evaluation functions Deep Blue’s brute-force alpha-beta search on specialized hardware
Go (2016-17) Expert heuristics, pattern libraries AlphaGo: deep learning + MCTS. Then AlphaGo Zero: zero human knowledge, pure self-play
Speech (2010s) Phoneme models, linguistic rules Hidden Markov Models + data, then deep learning
Vision (2012+) Edge detection, SIFT features, deformable parts CNNs learning features from data (AlexNet onward)
Language (2018+) Parse trees, grammar rules, NER pipelines Transformers + massive data + compute (GPT, BERT)

Kaplan et al. (2020)

The original scaling laws — OpenAI

  • Test loss follows power laws across 7+ orders of magnitude
  • Three axes: parameters (N), data (D), compute (C)
  • Small exponents — need ~10× more compute to halve the loss improvement
  • Diminishing returns, but returns nonetheless
// The power laws
L(N) ~ N^(-0.076)  // loss vs params
L(D) ~ D^(-0.095)  // loss vs data
L(C) ~ C^(-0.050)  // loss vs compute

What Kaplan got wrong

Compute allocation: N scales as C0.73, D scales as C0.27 — put most compute into bigger models, less into data.

This led the field astray for 2 years. GPT-3 (175B params, 300B tokens) was massively undertrained by later standards.

Root cause: counted non-embedding params only, studied small models, didn’t account for irreducible loss.

Chinchilla (2022)

DeepMind proves everyone was doing it wrong

  • Trained 400+ models from 70M to 16B params on 5B to 500B tokens
  • Core finding: model size and data should scale equally
  • Optimal ratio: ~20 tokens per parameter
  • Directly contradicted Kaplan’s 73/27 split favoring parameters

The Chinchilla model

70B params trained on 1.4T tokens (20:1 ratio). Same compute budget as Gopher (280B params, 300B tokens). 4× smaller, 4× more data, same FLOPs.

Chinchilla (70B) demolished larger models

ModelParamsMMLU
Chinchilla70B67.5%
Gopher280B60.0%
GPT-3175B~43%
Megatron-Turing NLG530B~56%

A 70B model beating a 530B model. Data matters as much as parameters.

Beyond Chinchilla — Overtraining for Inference

The evolution of tokens-per-parameter

ModelParamsTokensRatio
GPT-3 (2020)175B300B1.7:1
Chinchilla (2022)70B1.4T20:1
Llama 1 (2023)65B1.4T22:1
Llama 2 (2023)70B2T29:1
Llama 3 (2024)8B15T~200:1

Why overtrain?

Chinchilla optimizes for training compute. But if you serve the model to billions of inference requests, a smaller, more heavily trained model is cheaper to run.

Train once (fixed cost), serve forever (per-token cost). Minimize the per-token cost.

Research validated this: “When accounting for inference costs, you should train smaller and longer.” (Beyond Chinchilla-Optimal, 2024)

Emergent Abilities — Or a Mirage?

The claim (Wei et al., 2022)

  • Some abilities appear suddenly at a certain scale
  • 3-digit addition: GPT-3 6B = 1%, 13B = 8%, 175B = 80%
  • Near-random, then sudden jump — can’t be predicted by extrapolating smaller models
  • Other examples: word unscrambling, multi-step reasoning, chain-of-thought

The rebuttal (Schaeffer et al., NeurIPS 2023)

  • Emergence is an artifact of metric choice, not model behavior
  • Discontinuous metrics (Exact String Match) create apparent phase transitions
  • Replace with continuous metrics (Brier Score) → performance improves smoothly
  • 92% of “emergent” abilities on BIG-Bench appeared only under nonlinear metrics

Current consensus: Real capabilities do appear at scale, but the discontinuous jump narrative is largely a measurement artifact. Won Outstanding Paper at NeurIPS 2023.

When Bigger Is Worse

Inverse Scaling Prize (McKenzie et al., 2023)

  • Public contest to find tasks where larger models perform worse
  • Found 11 datasets with inverse scaling behavior

Four root causes

  • Memo Trap: prefer memorized phrases over following instructions
  • Bad patterns: imitating undesirable training data
  • Easy distractors: latching onto surface patterns
  • Misleading examples: overgeneralizing from few-shot demos

The U-shape twist

6 of 11 tasks showed U-shaped scaling when tested on even larger/better models — worse, then better again. Only 4 remained truly inverse.

Suggests RLHF and instruction-tuning eventually fix these failure modes.

Data quality > quantity

Microsoft’s Phi-1 (1.3B) outperformed models 10× larger trained on 100× more data on coding tasks. Secret: LLM-curated “textbook quality” synthetic data.

The New Scaling Axes

From training compute to inference compute

The Paradigm Shift

Old paradigm (2020–2023)

  • All scaling effort in training
  • Bigger models, more data, more FLOPs
  • GPT-4: ~2.15 × 1025 FLOPs, ~25K A100s for 90-100 days
  • Inference = “just run the model forward once”

New paradigm (2024+)

  • Test-time compute is a new scaling axis
  • Let a model “think longer” at inference
  • OpenAI’s o1 (Sept 2024): first major reasoning model
  • Inference demand: 118× training by 2026

OpenAI’s 2024 inference spend: $2.3 billion — 15× the training cost for GPT-4.5. Inference projected to claim 75% of AI compute by 2030.

Test-Time Compute Scaling

How “thinking longer” works

  • Extended chain-of-thought: Model generates long reasoning traces before answering
  • Tree search: Explore multiple reasoning paths, evaluate, select best
  • Repeated sampling + verification: Generate N answers, pick the best
  • Self-correction: Model checks its own work, backtracks on errors
  • Budget forcing: Control how many tokens the model uses for “thinking”

o1 → o3 on ARC-AGI

ModelScoreCost/Task
GPT-4o~5%$0.01
o132%~$5
o3 (low compute)75.7%~$200
o3 (high, 172×)87.5%~$20,000

Scaling test-time compute by 172× raises score from 75.7% to 87.5%. A new power law.

DeepSeek-R1 — Open Reasoning

  • Same V3 base (671B MoE), trained with pure RL
  • R1-Zero: developed chain-of-thought without human demonstrations
  • AIME 2024: 71% pass@1 (up from 15.6% for base), 86.7% with majority voting
  • Open-weight — democratized reasoning capability
  • RL training cost: only $294,000 on top of V3

The key insight

Dynamically allocates tokens: fewer for simple tasks, 10-100× more for complex ones. The model decides how hard to think.

s1: Reproducing the curve

Researchers showed that “budget forcing” (controlling thinking tokens) reproduces OpenAI’s published scaling curves. First open reproduction of test-time compute scaling laws (Jan 2025).

The Four Scaling Axes (2026)

1. Pre-training scale

More parameters, more data, more FLOPs. The original axis. Diminishing returns, but not exhausted.

2. Test-time compute

Let models think longer. Chain-of-thought, tree search, verification. Currently the hottest frontier.

3. Post-training (RL, RLHF, RLVR)

Reinforcement learning on top of base models. Synthetic data, reasoning traces. By 2027, most compute goes here, not pre-training.

4. Synthetic data & distillation

Large models generate training data for smaller ones. Fine-tuning datasets: Alpaca 52K → OpenThoughts 1M+ samples. Scaling the data itself.

The “Wall” Debate

“Scaling is over”

  • Ilya Sutskever (NeurIPS 2024): “The 2010s were the age of scaling, now we’re back in the age of wonder.”
  • Internet data is “the fossil fuel of AI” — finite
  • GPT-4.5, GPT-5: marginal improvements despite massive compute
  • OpenAI internally shifting from pre-training to RL post-training

“Scaling continues”

  • Dario Amodei: “Scaling is probably going to continue”
  • Sam Altman: “There is no wall”
  • Industry committed $7.8T to AI infra through 2030
  • Epoch AI: 5,000-300,000× GPT-4 scale possible by 2030
  • Synthetic + multimodal data expand the pool

The nuanced reality: Pre-training scaling is hitting diminishing returns. But scaling has shifted to new axes: post-training RL, test-time compute, synthetic data. It’s not “scaling hit a wall” — it’s “scaling found new dimensions.”

How Far Can We Go?

Epoch AI projections through 2030

  • Training runs of ~2×1029 FLOPs likely by 2030
  • 5,000–300,000× larger than GPT-4
  • GPT-4 exceeds GPT-2 in scale — 2030 models will exceed GPT-4 by the same margin

Four constraints

BottleneckLimit by 2030
Power supply4–16 GW per frontier run
Chip manufacturing~100M H100-equiv GPUs
Data availability400T–20Q tokens
Latency wall3×1030–1032 FLOPs

What 2026–2027 looks like

  • Inference: ~2/3 of all AI compute (up from 1/3 in 2023)
  • Post-training becomes the main game
  • Smaller, heavily trained models win for deployment (200:1 token/param is the new normal)
  • Data centers: ~$400B in 2026, rising to $1T annually

The Sweet Lesson

Is the Bitter Lesson the whole story?

Counter-arguments

  • Compute efficiency: Equivariant models with structured priors are more compute-efficient for a fixed budget
  • Sample efficiency: When data is scarce, inductive bias dramatically helps
  • Some problems are hopeless: Lattice field theory needs built-in symmetries
  • The Transformer itself is a bias: Attention = soft search. The Bitter Lesson doesn’t mean no structure

The resolution

“The sweet spot lies in using general, scalable methods within architectures that respect the fundamental geometry of the problem.”

The winning structures are the most general ones. Attention, MoE, RL — they work because they scale, not because they encode domain knowledge.

The Meta-Lesson

Scaling keeps finding new dimensions

2020 — Kaplan: “Scale parameters!” → GPT-3 (175B params, 300B tokens)
2022 — Chinchilla: “No, scale data equally!” → 20:1 token/param ratio becomes standard
2024 — Llama 3: “Actually, overtrain for inference!” → 200:1 ratio, optimize for serving cost
2024 — o1/R1: “Scale inference compute too!” → Test-time compute as new axis
2025 — Post-training: “Scale RL + synthetic data!” → Post-training delivers stronger returns than pre-training

At every stage, “just add more compute” (in whatever dimension) has won over clever engineering. The Bitter Lesson keeps proving itself.

Numbers Worth Knowing

FactNumber
Kaplan optimal allocation (params vs data)73% / 27% (wrong)
Chinchilla optimal ratio~20 tokens per parameter
Llama 3 ratio~200 tokens per parameter
GPT-4 training compute~2.15 × 1025 FLOPs
o3 high-compute ARC-AGI87.5% ($20K/task)
DeepSeek-R1 AIME 202479.8% (base: 15.6%)
Inference as % of AI compute (2026)~67%
Epoch: possible by 20302 × 1029 FLOPs (300,000× GPT-4)
“Emergent” abilities under nonlinear metrics92% (measurement artifact)
Inverse scaling tasks resolved by larger models6 of 11 (U-shaped)

Key Takeaways

  • Scaling laws are real — smooth power-law relationships across many orders of magnitude
  • Chinchilla changed the game — data matters as much as parameters. 20:1 ratio, not 2:1
  • But now we overtrain — optimize for inference cost, not training compute
  • Emergence is mostly measurement — smooth under continuous metrics
  • The Bitter Lesson holds — general methods + compute always win eventually
  • Scaling found new axes — test-time compute, post-training RL, synthetic data
  • Pre-training is diminishing — but the other axes are just beginning
  • The next axis matters most — whoever finds the next scaling dimension first has a massive advantage

General methods.
More compute.
New dimensions.

The bitter lesson keeps proving itself.

agent37.club