Multi-Head Latent Attention (MLA): the KV-cache breakthrough

The biggest cost in serving long-context models is the KV cache — the stored keys and values for every previous token, which grows with sequence length and dominates memory at inference. MLA compresses those keys and values into a small shared latent vector and reconstructs them on the fly, shrinking KV-cache memory by roughly an order of magnitude while keeping almost all of the quality. Less memory per request means more requests on the same hardware, which is a large part of why DeepSeek can serve a frontier model so cheaply.

• Standard attention stores full key/value tensors per token — memory grows linearly with context length
• MLA stores a compressed latent representation instead, cutting KV-cache size dramatically
• Lower memory per request raises serving throughput and slashes cost per token
• Later DeepSeek models add DeepSeek Sparse Attention (DSA) to push 1M-token context economically

DeepSeekMoE: fine-grained and shared experts

DeepSeek pairs MLA with a refined Mixture-of-Experts design. Rather than a few large experts, it uses many smaller, fine-grained experts plus a handful of always-on 'shared' experts that capture common knowledge. The router sends each token to a small set of specialists while the shared experts handle the general case. V3 has about 671 billion total parameters but activates only ~37 billion per token; V4-Pro scales to roughly 1.6 trillion total with ~49 billion active.

The release timeline: V2 to V4

Whatever happened to R2?: DeepSeek-R2 was never released. Reporting through 2025 attributed the delay to the CEO's dissatisfaction with its performance and to failed attempts to train on Huawei Ascend chips after US export controls tightened (training reverted to Nvidia; Ascend serves inference). In the end the standalone reasoning line was simply absorbed: V3.2-Speciale and V4's built-in Thinking mode are R2 in everything but name — mirroring the industry-wide merger of reasoning models into flagships.

Why $5.6M shocked the industry — and the caveats

When DeepSeek reported training V3 for about $5.6 million, the figure was startling against the hundreds of millions widely assumed for GPT-4-class models, and the R1 release in January 2025 was credited with a sharp sell-off in AI hardware stocks. The result was real, but a few caveats keep it honest.

• The $5.6M covers the final training run only — not research, failed runs, data, or staff
• Efficiency came from genuine engineering (MLA, fine-grained MoE, FP8 training), not a single trick
• Open MIT weights mean anyone can self-host — the disruption is as much about licensing as cost
• The hosted API routes through Chinese infrastructure, which raises data-residency and compliance questions for some buyers; self-hosting the open weights avoids that

Why it matters: DeepSeek proved that frontier capability is not gated behind a single lab's budget. Its architecture ideas — MLA and fine-grained MoE — have influenced the broader field, and its open weights gave the open-source ecosystem a genuine frontier-class option.

DeepSeek R1 and the reasoning era

DeepSeek R1 was the first widely available open model to match a closed reasoning model (OpenAI's o1) on hard math and coding. Crucially, DeepSeek published how it was trained — large-scale reinforcement learning on problems with verifiable answers, using a method called GRPO — which gave the open community a recipe to build reasoning models rather than just a model to download. That transparency accelerated the whole reasoning-model wave of 2025 and 2026.

• R1-Zero — the striking result: trained with RL only, no supervised fine-tuning step, the model spontaneously developed self-verification and backtracking. It was rough (language mixing, poor readability), but it proved reasoning emerges from reward alone.
• GRPO (Group Relative Policy Optimization): critic-free RL — sample a group of answers, score them against each other, reinforce the above-average ones. Cheaper and simpler than PPO with a learned critic; now standard vocabulary across the field.
• Distillation for everyone: R1's reasoning was distilled into six dense models from 1.5B to 70B (Qwen- and Llama-based, MIT-licensed) — putting usable reasoning on laptops within a week of release.
• Peer review milestone: R1's paper made the cover of Nature in September 2025 — the first frontier LLM to pass formal peer review — and disclosed that the RL stage cost just $294K on top of the base model.

Benchmarks and key specs

Why the numbers matter: R1's figures (from DeepSeek's own paper) matched OpenAI's o1 on math and reasoning, and V4-Pro lands within fractions of a point of closed frontier models on SWE-bench Verified — at a fraction of the per-token cost. All scores are vendor-reported pending independent replication.

Frequently asked questions

Did DeepSeek really train V3 for $5.6 million? That figure covers the final training run only — not research, failed runs, data, or staff. The efficiency is real, driven by MLA, fine-grained MoE, and FP8 training, but the headline number is not the all-in cost.

Is DeepSeek open source? Yes — V3, R1, and V4 ship under the permissive MIT license, so you can download the weights and self-host or fine-tune them, including commercially.

What is Multi-Head Latent Attention (MLA)? An attention design that compresses the KV cache into a small shared latent vector, cutting memory per request by roughly an order of magnitude and making long-context serving far cheaper.

Is DeepSeek safe for sensitive data? The hosted API routes through Chinese infrastructure, which raises data-residency and compliance questions for some organizations. Self-hosting the open MIT weights avoids that concern entirely.