• DeepSeek-R1 released — open-weight reasoning model
• Matched OpenAI o1 on AIME (79.8% vs 79.2%) and MATH-500 (97.3% vs 96.4%)
• DeepSeek app hit #1 on the App Store
• NVIDIA stock dropped 17% in one day — $600B in market cap evaporated
• Largest single-day loss for any company in stock market history

• Every token activates every parameter in the model
• GPT-4 (rumored ~1.8T params) — all params fire on every forward pass
• Llama 3.1 405B — 405 billion multiply-adds per token
• Linear scaling: 2× params = 2× compute per token

// Dense: every layer, every param
for layer in transformer:
    x = attention(x)
    x = FFN(x)  // ALL params used
    // 405B model = 405B ops/token

// Cost: O(params) per token
// Quality: great
// Efficiency: terrible

• Replace each FFN layer with N expert FFNs
• Add a router (gating network) that picks the top-K experts per token
• Only the selected experts compute — the rest sleep
• Total params grow with N, but active params stay fixed at K

// MoE: route to top-K experts
for layer in transformer:
    x = attention(x)

    // Router picks best experts
    scores = router(x)  // tiny MLP
    top_k = topk(scores, k=8)

    // Only K experts compute
    x = sum(w[i] * expert_i(x)
         for i in top_k)

// 671B total but ~37B active
// 18x parameter efficiency!

What the router must do

• A small MLP that takes a token’s hidden state and outputs scores for each expert
• Top-K experts are selected; their outputs are weighted-summed
• Must learn which experts specialize in what during training

The load balancing crisis

• Without intervention, routers collapse — all tokens go to 2-3 “popular” experts
• Most experts sit idle, wasting capacity
• Traditional fix: auxiliary loss that penalizes unbalanced routing
• Problem: auxiliary loss interferes with the actual training objective

// Traditional MoE routing
def route(x, num_experts, k):
    // Score each expert
    scores = softmax(W_gate @ x)

    // Pick top-k
    top_k_idx = topk(scores, k)
    weights = scores[top_k_idx]

    // Auxiliary loss to force balance
    // f_i = fraction of tokens to expert i
    // Want f_i ≈ 1/N for all experts
    aux_loss = N * sum(f_i * P_i)

    return top_k_idx, weights, aux_loss

// Too little aux_loss → collapse
// Too much aux_loss → hurts quality

• Old approach: Add a penalty term (auxiliary loss) to the training loss to encourage balanced expert usage. Always a compromise — hurts model quality.
• DeepSeek’s insight: Separate which expert gets selected from how much their output counts
• Selection uses a bias term that can be adjusted for balance without touching the loss function
• The expert’s contribution weight is learned independently through normal backprop

// DeepSeek-V3: decouple selection
// from contribution

// 1. Selection (with bias for balance)
score_i = sigmoid(W @ x)
gate_i  = score_i + bias_i
// bias_i adjusted to balance load
// NOT part of gradient computation

// 2. Contribution weight (learned)
weight_i = score_i / sum(scores)

// Selection ≠ weight
// Can balance routing without
// interfering with quality

• Standard attention: store Key and Value vectors for every past token
• For a 70B model with 128K context: ~40 GB just for KV cache
• KV cache grows linearly with sequence length and batch size
• This is often the bottleneck for serving at scale — you run out of GPU memory

Previous solutions

• GQA (Grouped Query Attention): share K/V heads across groups — Llama 2/3 use this
• MQA (Multi-Query Attention): single K/V head — aggressive, some quality loss

• Instead of storing full K and V matrices, compress them into a low-rank latent representation
• Store only the compressed vector in the KV cache
• At inference time, project back to full K/V dimensions
• Compression ratio: store ~1/4 to ~1/8 of the original size
• Performs better than standard MHA — the low-rank constraint acts as regularization

// Standard MHA
K = W_k @ x   // [d_model]
V = W_v @ x   // [d_model]
// cache: K + V per token per layer

// MLA: compress first
c = W_compress @ x  // [d_c] ≪ [d_model]
// cache: only c per token per layer!

// At attention time, decompress
K = W_uk @ c   // reconstruct K
V = W_uv @ c   // reconstruct V
// Then standard attention

// d_c ≈ d_model / 8
// Massive memory savings

The numbers

• 671B total parameters (+ 14B MTP module = 685B)
• 37B active parameters per token
• 61 transformer layers
• 256 routed experts + 1 shared expert per MoE layer
• 8 experts activated per token
• 128K context window
• MLA with compressed KV cache

Training details

• Trained on 14.8 trillion tokens
• Training hardware: 2,048 NVIDIA H800 GPUs
• Training time: ~2 months
• FP8 mixed precision training (first at this scale)
• Estimated cost: $5.576 million
• Pipeline parallelism across 2,048 GPUs with custom communication

• Standard LLM: predict the next 1 token
• MTP: predict the next N tokens simultaneously
• Forces the model to think further ahead during training
• Improves sample efficiency — more signal per training example
• Bonus: the MTP heads can be used for speculative decoding at inference time

// Standard: predict token t+1
loss = cross_entropy(
    model(tokens[:t]),
    tokens[t+1]
)

// MTP: predict t+1, t+2, ... t+n
for i in range(1, n+1):
    loss += cross_entropy(
        mtp_head_i(model(tokens[:t])),
        tokens[t+i]
    )

// At inference: MTP heads become
// draft model for speculative decode
// → 1.8x faster generation

What changed from V3 to R1

• Same base architecture (671B MoE)
• Trained with reinforcement learning to produce chain-of-thought reasoning
• Model “thinks aloud” before answering — similar to OpenAI o1
• First open-weight reasoning model at frontier level

Head-to-head vs OpenAI o1

What DeepSeek releases openly

• Full model weights (permissive license)
• Detailed technical reports with architecture specifics
• Training recipes and hyperparameters
• Distilled smaller models

Why give it away?

• DeepSeek isn’t an API company — no direct revenue model from models
• Open-sourcing builds reputation and attracts talent
• Parent company (High-Flyer) profits from using AI in trading, not selling it
• Geopolitical strategy: harder to restrict what’s already public

The bear case (Jan 27 narrative)

• If frontier models can be trained for $5.6M, why spend $400B on data centers?
• NVIDIA’s moat is undermined — you don’t need as many GPUs
• Hyperscalers are overbuilding

The bull case (Jevons)

• Cheaper AI = more AI deployment everywhere
• More models, more agents, more inference, more edge deployment
• Demand for compute increases when efficiency improves
• Historical: LEDs didn’t reduce electricity use — we just lit up everything

Threats

• Proves you can do frontier AI on fewer GPUs
• Shows export controls don’t fully constrain China (H800s are enough)
• MoE efficiency could reduce GPU requirements per model
• Custom communication libraries reduce dependency on NVLink

Reinforcements

• Inference demand grows faster than training shrinks
• MoE models need more memory (671B weights must be loaded)
• Every efficiency gain gets redeployed as more inference at scale
• NVIDIA’s CUDA ecosystem moat is deeper than hardware
• Blackwell B200 doubles inference performance — demand remains huge

Memory footprint

• All 671B parameters must be loaded into memory
• Even though only 37B are active per token, the full model sits in VRAM/RAM
• MoE models are harder to run locally than dense models of equal quality

Routing overhead

• Expert selection adds latency
• Scattered memory access patterns are GPU-unfriendly
• If routing isn’t perfectly balanced, some GPUs idle while others bottleneck

Training complexity

• Distributing experts across GPUs requires careful pipeline design
• Communication patterns are more complex than dense models
• Load balancing is hard to get right (hence DeepSeek’s innovation)

Inference serving

• Expert parallelism requires all-to-all communication
• Google’s Switch Transformer paper: “making a serving system for such a model would require significant engineering”
• Batch efficiency drops when different tokens route to different experts

• MoE decouples knowledge from compute — massive parameter counts with fraction of the FLOPs per token
• DeepSeek innovated at every layer: fine-grained experts, shared experts, MLA, aux-loss-free routing, multi-token prediction, FP8 training
• $5.6M to match GPT-4 — the cost of training frontier models is dropping faster than anyone expected

• Open-weight matters — detailed technical reports let the whole field learn and build on top
• Jevons Paradox applies — efficiency doesn’t reduce compute demand, it increases adoption
• Constraints breed innovation — H800 export restrictions didn’t stop DeepSeek, they forced creative engineering
• MoE is now the default for frontier models — dense architectures are the exception