• A $2,000 GPU can run models matching GPT-3.5 quality
• Distilled 7B models rival GPT-4 on specific tasks
• GPT-4-equivalent API cost: $0.40/M tokens (was $20 in 2022)
• The gap between local and frontier closes every quarter

• Models are trained in FP16 (16-bit floating point) — each weight uses 2 bytes
• A 7B param model = 14 GB in FP16
• Quantization reduces precision: 16-bit → 8-bit → 4-bit → 2-bit
• Each halving of bits ≈ halves the model size
• Key insight: most weights don’t need full precision — the important ones are sparse

• Created August 2023 by Georgi Gerganov (llama.cpp)
• Single binary file: model weights + tokenizer + architecture + quant params
• Runs on everything: CPUs, GPUs, Apple Silicon, ARM
• Memory-mapped (mmap): OS loads only what’s needed
• Native format for Ollama, LM Studio, GPT4All

Quantization levels (7B model)

# That's it. One command.
ollama run llama3.2

# Downloads the GGUF, loads it,
# starts chatting. Q4_K_M by default.
# Works on Mac, Linux, Windows.

# Or pick your quant:
ollama run llama3.2:8b-q5_K_M

# API compatible with OpenAI:
curl http://localhost:11434/v1/chat/completions \
  -d '{"model":"llama3.2","messages":[...]}'

What 1.58 bits means

• Weights are ternary: {-1, 0, +1}
• Each weight needs log₂(3) = 1.58 bits
• Multiplications become simple additions/subtractions
• No floating-point multiply needed

Microsoft’s BitNet b1.58 2B4T

• 2 billion params, trained on 4 trillion tokens
• Comparable to leading FP16 2B models
• Released April 2025 as first open 1-bit LLM

Efficiency gains

Teacher → Student

• Large “teacher” model generates training signals
• Small “student” learns to replicate not just answers but probability distributions
• Transfers “dark knowledge” — nuanced relationships, reasoning patterns, confidence

Two approaches

• Output distillation: match teacher’s logits/probabilities
• Data distillation: teacher generates high-quality synthetic training data

Modern LLM distillation mostly uses approach #2: generate massive reasoning datasets, fine-tune student.

# Modern distillation pipeline

# 1. Teacher generates reasoning
teacher = DeepSeekR1("671B")
for problem in dataset:
    trace = teacher.generate(
        problem,
        show_reasoning=True
    )
    training_data.append({
        "input": problem,
        "output": trace
    })

# 2. Student learns from traces
student = Qwen2_5("7B")
student.finetune(training_data)

# Result: 7B model with 671B reasoning

DeepSeek-R1 Distilled Models

A 32B distilled model outperforms OpenAI o1-mini. And it’s open-weight.

Microsoft Phi Series

• Phi-1 (1.3B): outperformed 13B models on coding with “textbook quality” synthetic data
• Phi-4 (14B): 84.8% MMLU. Beat GPT-4o on MATH and GPQA
• Phi-4-reasoning-plus: 77.7% on AIME 2025, beating DeepSeek-R1-Distill-70B (5× its size)

Apple Silicon

Memory bandwidth is the limiting factor, not compute.

NVIDIA Consumer GPUs

Local wins when

• Privacy: Data never leaves device (HIPAA, legal, finance)
• Latency: Zero network round-trip (sub-ms vs 200–500ms)
• Cost at scale: >2M tokens/day, hardware amortizes fast
• Offline: Airplanes, air-gapped networks, rural areas
• No rate limits: As fast as your hardware allows
• Customization: Fine-tune freely, no ToS restrictions

API wins when

• Quality: Frontier models still significantly better on hard reasoning
• Ease: One API key vs managing hardware
• Low volume: <100K tokens/day = API is cheaper
• Always current: Provider handles updates
• Multimodal: Latest vision/audio ships to APIs first
• Scale elasticity: Handle traffic spikes

At 5M tokens/day (team usage)

Hybrid approach: local for 80% routine tasks, API for 20% complex. Typical savings: 60–70%.

Break-even thresholds

Tokens per second (8B Q4 model)

# Install Ollama
brew install ollama   # macOS
# or: curl -fsSL https://ollama.com/install.sh | sh

# Run a model
ollama run llama3.2       # 8B, general
ollama run deepseek-r1:8b  # reasoning
ollama run qwen2.5-coder   # coding
ollama run gemma3:12b      # multimodal

# Use as API (OpenAI-compatible)
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.2",
       "messages":[{"role":"user",
       "content":"Hello!"}]}'

Recommended starter setups

• Quantization is nearly free — INT4 retains 98% quality at 25% the size
• GGUF + Ollama = running AI locally is trivial — one command, any hardware
• BitNet points to the future — 100B models on CPUs at reading speed
• Distillation transfers reasoning — 32B model beats o1-mini, costs $450 to create

• The 7B–14B range is the sweet spot — runs on any GPU, quality improving fast
• Bandwidth > FLOPS for inference — buy the chip with more memory bandwidth
• Hybrid is the answer — local for 80% of tasks, API for the hard 20%
• Cost drops 50× in 2 years — and still falling