Introduction

DeepSeek-V3.2 pairs architectural changes with a large-scale reinforcement learning and environment synthesis pipeline to make long-context reasoning and multi-turn tool use affordable at scale. It modifies how attention reads a long conversation history, stabilizes off-policy reinforcement learning across a massive mixture-of-experts network, and trains on thousands of synthetic agent environments where actions can be verified automatically.

This book builds that technical story from the ground up. Each chapter introduces the required mathematical and systems context, walks through one subsystem in detail, and connects its design choices to the rest of the stack. You can read the chapters sequentially or jump directly to a specific subsystem.

Our primary source is DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (arXiv:2512.02556v1, submitted 2 December 2025), supplemented by revision-pinned configuration, inference, and encoding files from the official open-weights release. Where the paper leaves training hyperparameters or systems numbers unstated, we note those gaps directly and derive what we can from the published equations and checkpoint specifications.

Contents

Reading paths

If you are focused on model architecture, read Chapters 2–7 and 22. For reinforcement learning, read Chapters 8–12 and 21. For agent training and tool use, read Chapters 13–18, 20, and 23. For benchmarks and deployment trade-offs, read Chapters 1, 7, 19–21, and 24. Chapter 25 serves as a technical glossary, equation index, and file atlas.

Whichever path you take, start with Chapter 1 to keep the four model variants in the paper straight: DeepSeek-V3.1-Terminus, DeepSeek-V3.2-Exp, the final DeepSeek-V3.2, and DeepSeek-V3.2-Speciale represent distinct training stages and capability trade-offs.

Source and numerical conventions

Chapter source notes cite the corresponding report sections, TeX files, and official repository artifacts preserved under ../_sources/. The TeX source archive is stored at ../2512.02556v1-source.tar.gz with ../main.tex as its entry point, and original figures are retained under ../figures/ alongside rasterized versions in assets/.

Unless noted otherwise, K, M, and B denote powers of 1,000. In the continued-training token calculations, however, the paper's sequence counts and token totals only reconcile if 128K denotes 131,072 tokens ($128 \times 1024$). We retain the paper's 128K shorthand when discussing the context window and note where the configuration files specify different limits (examined in Chapters 3 and 22). Memory capacities use GB for $10^9$ bytes and GiB for $2^{30}$ bytes. Differences between percentage scores are stated in percentage points, while Codeforces comparisons use rating points.

1. The Problem and the Model Family

A language model that answers short prompts fluently can still fall apart on multi-step engineering or research problems. Fixing a software bug requires inspecting source files, writing a patch, running the test suite, and recovering when an initial hypothesis fails. Answering an open-ended research question requires issuing multiple searches, cross-checking sources, and reconciling conflicting claims. Proving a competition math theorem often takes thousands of tokens of Scratchpad exploration before a viable proof strategy emerges.

Tasks like these strain three parts of the stack simultaneously: the model needs strong multi-step reasoning habits, a large reinforcement-learning budget to practice them, and an attention mechanism that can read a long, growing context without blowing up inference cost. DeepSeek-V3.2 tackles all three together, treating sparse attention, RL optimization stability, and synthetic agent environments as mutually reinforcing pieces of a single design.

Why long reasoning changes the economics

Because an autoregressive transformer generates one token at a time conditioned on the entire preceding sequence, a 20,000-token reasoning trace requires forty times as many decode passes as a 500-token reply—and each pass attends over a longer history than the last. Tool use compounds this growth: every file dump, compiler trace, or search snippet returned by an external environment is appended directly to the context window. Even if the model writes terse action calls, verbose tool outputs rapidly fill a 128K window.

Under dense causal attention, every query token compares against every preceding key token. Prefilling a long prompt scales quadratically with sequence length $L$, while decoding each new token requires scanning a cache whose length grows linearly with every step. That cost hits twice: once when serving long-horizon user requests in production, and again during reinforcement learning, when the training cluster must sample thousands of long rollouts per update.

To cut the cost of reading long histories, the paper introduces DeepSeek Sparse Attention (DSA). A lightweight module called the lightning indexer scores all past positions, and the main Multi-head Latent Attention (MLA) kernel loads only the top 2,048 token entries per query. The indexer itself still scans the full prefix—a crucial systems nuance we analyze in Chapter 7—but the heavy main-attention read drops from $O(L^2)$ to $O(Lk)$ during prefill.

Why architecture alone cannot produce an agent

Making long contexts cheaper to read does not automatically teach a model how to use them. Given a fast 128K context window, a poorly trained assistant will still hallucinate tool arguments, drop constraints halfway through a task, or declare victory without verifying its output.

The authors trace the gap between open-weights models and frontier proprietary systems to two bottlenecks: insufficient post-training RL compute and a shortage of diverse, verifiable agent environments. In DeepSeek-V3.2, the post-training RL budget exceeds 10% of the total pre-training cost (the report states this as a relative share without disclosing raw GPU-hours or dollar figures). Lowering attention cost makes long-horizon rollouts cheaper to sample, while stabilizing off-policy GRPO updates lets the model absorb a much larger RL compute budget across synthetic coding, search, and tool-use environments.

Keeping the four model variants straight

The report tracks four distinct checkpoints across its experiments:

Separating these checkpoints prevents a common misreading of the benchmark tables. The architectural ablation compares DeepSeek-V3.1-Terminus against DeepSeek-V3.2-Exp, which shares the Terminus training style after continued training to install DSA. The jump in reasoning and agent scores between V3.2-Exp and the final DeepSeek-V3.2 comes from the expanded post-training pipeline—specialist distillation, mixed RL, and synthetic agent environments—not from sparse attention itself.

DeepSeek-V3.2-Speciale branches off the same post-training pipeline but trains exclusively on reasoning data with a weaker length penalty, plus the dataset and proof-verification reward system from DeepSeekMath-V2. According to the official model card, Speciale omits tool-calling support altogether; it is built to maximize competition reasoning performance at the cost of much longer token trajectories, not to serve as a drop-in replacement for the general model.

Three distinct forms of sparsity and truncation

Because DeepSeek-V3.2 uses selection mechanisms at three completely different stages of computation, it helps to separate their scopes upfront:

1. Expert sparsity (MoE): Each token routes through 8 out of 256 routed feed-forward experts per layer (plus 1 shared expert), bounding the active FFN parameter count per token.
2. Attention sparsity (DSA): Each query token reads 2,048 historical key-value entries out of a prefix of up to 128K tokens, bounding the main attention cost.
3. Sampling truncation (top-$p$ / top-$k$): During rollout generation, the sampler zeros out low-probability vocabulary tokens before drawing the next token, which shapes rollout diversity and must be preserved via sampling masks during off-policy RL updates.

Although the paper discusses masks and top-$k$ selection in all three contexts, they operate on different axes: feed-forward modules, sequence positions, and vocabulary logits.

What the open release includes

The weights and inference code for DeepSeek-V3.2 are released under the MIT license, allowing direct inspection of the layer definitions, quantization formats, and chat templates, as well as local deployment. As with most open-weights frontier releases, however, the full pre-training corpus, distributed RL orchestration code, reward models, and synthetic task generators remain internal. Throughout this book, we cross-check the paper's equations against the released model.py and config.json artifacts so you can see exactly where the paper's high-level description meets the production tensor shapes.

Reading the paper's headline claims

When evaluating the report's three headline claims—frontier parity, Olympiad gold-medal performance, and reduced attention complexity—it pays to look closely at the underlying unit of measurement:

• Parity with frontier models: V3.2 matches or approaches GPT-5-High on several math and reasoning benchmarks, while trailing on broad factual knowledge and complex multi-turn tool benchmarks. Speciale overtakes frontier baselines on several reasoning tests, but uses substantially more output tokens per problem and drops tool support.
• Olympiad gold-medal scores: On IMO, CMO, ICPC, and IOI evaluations, the reported scores reflect a complete test-time system that includes parallel candidate sampling, filtering, and—for mathematical proofs—iterative self-verification, rather than single-pass greedy decoding.
• End-to-end task economics: Even if sparse attention speeds up per-token prefill and decode on long sequences, a model that generates twice as many reasoning tokens or executes dozens of tool turns can still cost more per completed task. The meaningful systems metric is the total compute spent to solve a problem, not the latency of a single attention kernel in isolation.

Sources: Report §1 and §3 (sections/01-0-intro.tex, sections/02-post.tex); official model card at the preserved revision. Continue with the foundations.

2. The Foundations the Report Assumes

Before diving into DeepSeek-V3.2's specific modifications, this chapter establishes the core notation and mechanics the report takes for granted: autoregressive tokenization, pre-norm transformer blocks, causal attention and KV caching, and policy-gradient reinforcement learning over multi-turn agent trajectories.

Tokens and next-token probabilities

A tokenizer maps raw text into a sequence of integer IDs drawn from a fixed vocabulary $\mathcal V$. Depending on the byte-pair or subword vocabulary, a single token might represent a full word, a subword stem, punctuation, whitespace, or a reserved control symbol such as <｜tool▁call▁begin｜>. Token counts therefore vary across languages and formatting styles for the exact same semantic content.

At the input layer, each token ID is mapped to a learned embedding vector. A stack of transformer layers refines these vectors into contextual representations, and a final linear head projects the top-layer hidden state into one logit $z_v$ per vocabulary item. Applying a softmax yields the next-token probability distribution conditioned on the prefix $x_{\le t}$:

During generation, a sampler selects the next token according to a decoding rule (such as temperature scaling followed by top-$p$ truncation), appends it to the prefix, and runs the next step. Whether the emitted tokens form a natural-language paragraph or a JSON tool invocation makes no difference to the underlying probability model; an external runtime parser watches the output stream and intercepts structured tool calls when a stop sequence appears.

Transformer blocks: communication vs. local computation

Each transformer layer alternates between two operations wrapped in residual connections and root-mean-square normalization (RMSNorm):

Attention moves information across sequence positions, letting token $t$ read states from tokens $s \le t$. The feed-forward network (FFN) transforms features within each position independently, operating on a single token's residual vector after attention has mixed in contextual signals. In a Mixture-of-Experts (MoE) layer, that single dense FFN is replaced by a bank of parallel expert FFNs controlled by a learned router.

Queries, keys, and values

Inside a single attention head, position $t$ projects its normalized hidden state into a query vector $q_t \in \mathbb R^{d_k}$, while every historical position $s \le t$ projects its state into a key vector $k_s \in \mathbb R^{d_k}$ and a value vector $v_s \in \mathbb R^{d_v}$. The scaled dot product between $q_t$ and $k_s$ scores their compatibility, and a softmax converts those scores into convex combination weights over the value vectors:

In short, the query specifies what the current position is looking for, the keys determine where to attend, and the values supply what gets read. Multi-head attention runs dozens of these heads in parallel with distinct learned projections so a token can simultaneously attend to different positions and subspaces, then concatenates and projects the head outputs back into the residual stream.

Causal masking and the quadratic triangle

Because an autoregressive model cannot peek at future tokens when predicting token $t+1$, a causal mask sets attention logits for all positions $s > t$ to $-\infty$, driving their softmax weights to zero. Over a prompt of length $L$, the number of valid query-key pairs $(t, s)$ with $1 \le s \le t \le L$ is the lower triangular sum:

Thus, causal masking cuts the work in half relative to bidirectional attention, but the pair count still scales as $\Theta(L^2)$.

Prefill, decode, and the KV cache

Inference splits into two distinct execution regimes:

• Prefill ingests an entire prompt of length $L$ at once. Because all prompt tokens are already known, the hardware can evaluate all $L$ queries in parallel using dense matrix multiplications, subject to the lower-triangular causal mask.
• Decode generates new tokens one at a time. Because token $t+1$ must be sampled before token $t+2$ can be computed, decode steps are inherently sequential and typically bottlenecked by memory bandwidth rather than raw arithmetic throughput.

To avoid recomputing keys and values for every historical token at every decode step, inference engines maintain a KV cache in GPU memory. Standard multi-head attention caches explicit $k_s$ and $v_s$ tensors for every head, which quickly consumes tens of gigabytes on long sequences. DeepSeek's Multi-head Latent Attention (MLA) compresses that cache into a shared low-rank latent vector per token, while DeepSeek Sparse Attention (DSA) reduces how many cached tokens each query actually reads. MLA shrinks the storage footprint of the history; DSA shrinks the read volume per step.

From next-token cross-entropy to post-training

Pre-training minimizes the standard cross-entropy loss over a massive text corpus:

Predicting the next token over trillions of words teaches the base model syntax, world knowledge, code idioms, and reasoning patterns, but it does not align the model to follow multi-constraint user instructions or recover from runtime errors in an interactive environment. Post-training bridges that gap through three mechanisms:

1. Supervised fine-tuning (SFT) trains on curated prompt-response demonstrations.
2. Knowledge distillation transfers behavior from specialized domain teacher checkpoints into a single general student checkpoint.
3. Reinforcement learning (RL) samples candidate trajectories from the model, scores them with rule-based verifiers or reward models, and updates the weights to increase the probability of high-scoring trajectories.

Policies, agent trajectories, and advantages

In RL terminology, the language model is a stochastic policy $\pi_\theta$ parameterized by weights $\theta$. For a single-turn prompt $q$, the probability of generating an output sequence $o = (o_1, \dots, o_{|o|})$ factors autoregressively over its tokens:

In a multi-turn agent environment, a trajectory interleaves model-generated tokens (internal reasoning and tool calls) with environment observations (command outputs, search results, or error traces). Although the model conditions on the entire interleaved transcript when generating its next action, only the tokens actually sampled from $\pi_\theta$ are actions of the policy; environment observations must be masked out when computing policy gradients.

A verifier or reward function assigns a scalar reward to the completed trajectory, and Group Relative Policy Optimization (GRPO, detailed in Chapter 9) converts a group of rewards for the same prompt into a baseline-subtracted advantage. If four sampled rollouts receive binary rewards $[1, 1, 0, 0]$, their group mean is $0.5$, yielding mean-centered advantages $[0.5, 0.5, -0.5, -0.5]$ that push probability mass toward the successful rollouts and away from the failures.

Because a verifier only needs to check a final answer or run a unit test suite rather than grade every intermediate step, tasks that are hard to solve but cheap to verify—such as software patches, math proofs, and constraint-satisfaction puzzles—form the backbone of scalable RL training. Even so, an RL policy is only as good as its verifier: if a test suite has gaps or a constraint checker has a bug, the optimizer will happily exploit it.

Sources: Transformer and RL policy formulation in Report §§2–3; released model.py (MLA, Block, and Transformer classes) in the pinned official implementation. Continue with the backbone and experts.

3. Inside the Inherited Backbone

DeepSeek-V3.2 builds directly on the Mixture-of-Experts (MoE) transformer backbone introduced in DeepSeek-V3 and refined through V3.1-Terminus. While sparse attention is the headline architectural addition in V3.2, attention is only one piece of the forward pass: every token still flows through embeddings, RMSNorm layers, query/key/value projections, routed feed-forward experts, and a 129,280-entry vocabulary head.

Inspecting the revision-pinned config.json and model.py files in the official repository anchors the paper's equations in concrete tensor dimensions.

Architectural dimensions in the released checkpoint

The Hugging Face config.json also lists num_nextn_predict_layers: 1, which carries over the auxiliary Multi-Token Prediction (MTP) head from DeepSeek-V3. During standard autoregressive inference in model.py, the model executes the 61 main layers; using the MTP head for speculative decoding requires dedicated runtime support outside the minimal reference script.

Why 128 heads do not mean a 128-head KV cache

A glance at num_key_value_heads: 128 in config.json can mislead off-the-shelf memory calculators designed for standard Grouped-Query Attention (GQA). In a conventional transformer with 128 KV heads, caching per-head keys and values for every token would require storing $128 \times (192 + 128) = 40{,}960$ elements per token per layer.

DeepSeek-V3.2 uses Multi-head Latent Attention (MLA), which never materializes per-head key and value caches in memory. Instead, each layer caches a single shared 512-dimensional latent vector alongside a 64-dimensional rotary positional key—just 576 elements per token per layer—and folds the per-head expansion matrices into the query and output projections at runtime. Chapter 4 walks through this matrix absorption step by step.

Inside an MoE layer: SwiGLU experts and group-limited routing

After the first three layers (which use standard dense SwiGLU feed-forward networks with intermediate dimension 18,432), the remaining 58 layers replace the dense FFN with an MoE module. Each individual expert $e$ is a gated SwiGLU network mapping $\mathbb R^{7168} \to \mathbb R^{7168}$ through a 2,048-dimensional bottleneck:

For each token $t$, a learned router selects a subset $\mathcal E_t$ of 8 routed experts out of 256, scales their outputs by gating weights $g_{t,e}$, and adds the result to an always-active shared expert:

In the reference implementation (Gate in model.py), routing follows a two-stage group-limited procedure:

1. Unbiased affinity scoring: The router computes a linear projection followed by a sigmoid activation, producing raw affinity scores $s_{t,e} \in (0, 1)$ across all 256 experts.
2. Biased group and top-$k$ selection: A learned load-balancing bias $b_e$ is added to form selection scores $s'_{t,e} = s_{t,e} + b_e$. The 256 experts are partitioned into 8 groups of 32; each group is scored by summing its top 2 biased expert scores, the top 4 groups are retained, and the top 8 experts overall are picked from those 4 groups.
3. Unbiased weight normalization: Once the 8 expert IDs $\mathcal E_t$ are chosen, their combination weights $g_{t,e}$ are computed by normalizing the original unbiased sigmoid scores $s_{t,e}$ across the 8 selected experts and multiplying by route_scale = 2.5.

Separating biased discrete selection from unbiased continuous weighting keeps load balancing from distorting the magnitude of expert contributions. At the same time, because top-$k$ selection is a discrete argmax operation, tiny floating-point shifts between rollout generation and training replay can flip which experts cross the threshold—the exact instability that motivates the Keep Routing replay mechanism covered in Chapter 11.

Parameter accounting: 671B total vs. 37B active

The DeepSeek-V3 backbone is cited at roughly 671B total parameters with 37B activated per token (adding the V3.2 lightning indexer changes the total by less than 0.2%). We can verify how the MoE layers dominate that budget directly from the tensor shapes.

Ignoring normalization vectors and router biases, a single expert contains three matrices of shape $7168 \times 2048$:

Across the 58 MoE layers, the 256 routed experts per layer account for:

When processing a single token, however, each MoE layer evaluates only 8 routed experts plus 1 shared expert, activating:

Adding the 3 initial dense FFN layers, the 61 MLA blocks, the embedding matrix, the LM output head, and the MTP module brings the per-token active count to roughly 37B parameters and the stored total to roughly 671B.

While a single token only pays the FLOP cost of 37B parameters, a serving cluster cannot fit the model into the VRAM of a 37B dense checkpoint: across a batch of tokens, almost all 256 experts in every layer will be touched, so the full 671B weight footprint must stay resident in high-bandwidth memory across the GPU cluster.

MoE layer experts vs. post-training domain specialists

The paper uses the word "specialist" for two completely unrelated mechanisms at different scales:

Domain specialists exist only during the post-training pipeline (Chapter 8); at inference time, a single unified checkpoint serves all tasks, and its internal MoE routers pick combinations of FFN submodules token by token without any explicit notion of task labels.

Reconciling the three context-length numbers in the release

If you inspect the paper alongside the repository files, you will encounter three different sequence-length constants:

• 128K (131,072 tokens): The context window used throughout the paper's continued pre-training, RL rollouts, and long-context evaluations.
• 163,840 tokens: The max_position_embeddings field in the Hugging Face config.json, parameterized with YaRN rotary scaling (factor: 40 over original_max_position_embeddings: 4096).
• 16,384 tokens: The default max_seq_len hardcoded in the minimal inference/model.py demo script so that a reference test run does not pre-allocate a massive KV cache by default.

Throughout our analysis, we use 128K when discussing the paper's training and evaluation regime, and write out explicit token counts ($131{,}072$ vs. $128{,}000$) whenever we compute exact FLOPs or memory footprints.

Sources: Report §2 (sections/01-method.tex); released config.json; pinned inference/config_671B_v3.2.json and inference/model.py (Gate, MoE, and Block classes). Continue with Multi-head Latent Attention.

4. Multi-Head Latent Attention

Multi-Head Latent Attention (MLA) compresses the per-token attention state stored in the KV cache. DeepSeek-V3.2 inherits MLA directly from the V3 backbone and wraps sparse token selection around it. Understanding how MLA structures its cache explains both what DeepSeek Sparse Attention (DSA) actually selects and why sharing each selected token entry across all 128 query heads maps so cleanly onto GPU memory hierarchies.

The core algebraic insight of MLA is straightforward: instead of caching explicit per-head key and value vectors for every past token, the layer caches a single low-rank latent vector and reconstructs per-head keys and values through linear projections. Because those reconstructions are linear, the up-projection matrices can be absorbed directly into the query and output projections during decoding.

Compressing keys and values into a shared latent

Let $h_s \in \mathbb R^d$ ($d = 7168$) be the hidden state at historical position $s$. MLA projects $h_s$ down into a compressed key-value latent vector $c_s^{KV} \in \mathbb R^{d_c}$ ($d_c = 512$) and applies RMSNorm:

For each attention head $i \in \{1, \dots, 128\}$, learned up-projection matrices $W_i^K \in \mathbb R^{128 \times 512}$ and $W_i^V \in \mathbb R^{128 \times 512}$ map that shared 512-dimensional latent into head-specific content keys and values:

Even though all 128 heads read from the same compressed latent $c_s^{KV}$, each head applies its own learned subspace projection $W_i^K$ and $W_i^V$, preserving distinct per-head attention patterns. Queries undergo a similar low-rank bottleneck ($7168 \to 1536 \to 128 \text{ heads}$) to save parameter and activation memory, though only the key-value latent $c_s^{KV}$ persists in the historical cache across decoding steps.

Absorbing the key up-projection into the query

During decoding, explicitly multiplying $W_i^K c_s^{KV}$ for every historical token $s$ and every head $i$ would defeat the bandwidth savings of caching a compact latent. Instead, look at the content compatibility score between query head $i$ at current position $t$ and historical position $s$:

By associativity of matrix multiplication, we can group $W_i^K$ with the current query vector instead of the historical latent:

For a single new decode token $t$, the model multiplies $q_{t,i}^C \in \mathbb R^{128}$ by $(W_i^K)^\top$ once to form an absorbed query $q_{t,i}^A \in \mathbb R^{512}$ in latent space. All 128 absorbed query heads then compute dot products against the exact same cached vector $c_s^{KV} \in \mathbb R^{512}$, eliminating per-head key materialization across the historical sequence.

Absorbing the value up-projection into the output

The same associativity trick eliminates per-head value reconstruction. Given softmax attention weights $a_{t,s,i}$ for head $i$, the head output is:

Factoring the position-independent matrix $W_i^V$ outside the sum over $s$ gives:

Instead of expanding $c_s^{KV}$ into 128 distinct value vectors at every historical position $s$, the attention kernel accumulates a weighted sum of the shared 512-dimensional latents $c_s^{KV}$ for each head, and only multiplies by $W_i^V$ once per head after the sequence reduction is complete (or fuses $W_i^V$ directly into the final output projection $W^O$).

Why rotary position embeddings require a separate 64-dim path

Relative positional encoding via Rotary Position Embedding (RoPE) applies a position-dependent rotation matrix $R_t$ to queries and $R_s$ to keys so that their inner product depends on the relative offset $t - s$. If we tried to apply RoPE inside the compressed latent before multiplying by $W_i^K$, the position-dependent matrix $R_s$ would sit between $W_i^K$ and $c_s^{KV}$, preventing us from absorbing $(W_i^K)^\top$ into the query vector at step $t$.

MLA solves this by decoupling content addressing from positional addressing. Alongside the 512-dimensional content latent $c_s^{KV}$, each token produces a single shared 64-dimensional positional key $k_s^R \in \mathbb R^{64}$ rotated by $R_s$, while each query head produces a 64-dimensional positional query $q_{t,i}^R \in \mathbb R^{64}$ rotated by $R_t$. The full attention logit adds the two inner products:

followed by softmax scaling ($\frac{1}{\sqrt{128 + 64}}$) and causal masking. Each historical token therefore caches $512 + 64 = 576$ numbers per layer—shared across all 128 query heads. (Chapter 22 covers the exact runtime scale factors and RMSNorm placement in model.py.)

MHA mode vs. MQA mode execution

Appendix A of the report contrasts the two ways an engine can execute the exact same MLA layer:

Reproduced from Report Appendix A (Figure 7). Both diagrams compute the same mathematical function, arranged for different compute-to-memory-bandwidth regimes.

• MHA mode (unabsorbed): Explicitly projects $c_s^{KV}$ into 128 per-head keys and values before running standard Multi-Head Attention. During dense prefill or training over long prompts, many queries are processed simultaneously, arithmetic intensity is already high, and avoiding the wider 512-dim per-head dot product keeps FLOPs lower.
• MQA mode (absorbed): Keeps $c_s^{KV}$ unexpanded in memory and absorbs $W_i^K$ and $W_i^V$ into the query and output paths. Structurally, the attention kernel now looks like Multi-Query Attention (MQA), where 128 query heads attend to a single shared key-value head of dimension 576 ($c_s^{KV} \Vert k_s^R$) and accumulate values of dimension 512 ($c_s^{KV}$).

In DeepSeek-V3.1-Terminus, the engine runs MHA mode for training and prefill, and switches to MQA mode for decoding. In DeepSeek-V3.2, DSA is built specifically around MQA mode: when the lightning indexer selects 2,048 token indices for query $t$, a sparse MQA kernel gathers each selected 576-element entry once from HBM and reuses it across all 128 query heads in SRAM. Only on short-sequence prefills—where dense matrix kernels outperform sparse gather kernels—does the engine fall back to a masked MHA kernel to simulate the sparse selection pattern.

Sizing the MLA cache at 128K

To see the memory footprint in concrete terms, consider a single sequence of $L = 131{,}072$ tokens across all 61 layers. If both the 512-dimensional latent $c_s^{KV}$ and the 64-dimensional RoPE key $k_s^R$ are stored in BF16 (2 bytes per element), the MLA cache for one sequence requires:

By comparison, an un-compressed 128-head BF16 KV cache with head dimensions $192 + 128 = 320$ would require $61 \times 131{,}072 \times 128 \times 320 \times 2 \approx 655\text{ GB}$ per sequence. Furthermore, as noted in the model.py source comments, production deployments quantize the KV cache to FP8, cutting the MLA footprint roughly in half again (though DSA adds a small separate cache for indexer keys, which we account for in Chapter 7).

Why MLA and DSA fit together

MLA compresses each token's feature vector along the channel axis without merging neighboring tokens along the time axis; every token position $s$ still has its own distinct cache slot $c_s^{KV}$. That design gives DSA clean, fine-grained addressability—it can cherry-pick individual token positions rather than coarse multi-token blocks—while MLA's shared-latent MQA layout ensures that every irregular memory gather from the KV cache feeds all 128 query heads at once.

Sources: Report §2.1 and Appendix A (sections/01-method.tex, sections/appendix.tex); pinned inference/model.py (MLA class) in the official inference repository. Continue with the lightning indexer.

5. The Lightning Indexer and Sparse Selection

DeepSeek Sparse Attention (DSA) splits attention into a cheap candidate-ranking pass and an expressive latent-reading pass. First, the lightning indexer scans the prefix and scores every historical token position for the current query. Next, a top-$k$ selector picks the 2,048 highest-scoring positions, and Multi-head Latent Attention (MLA) runs only over those selected entries.

Because the ranking is computed freshly at every layer and every query position, token selection is fully dynamic and content-dependent rather than constrained to fixed stride patterns or local sliding windows.

DSA architecture overview

Reproduced from Report Figure 1. Green modules denote the lightning indexer and top-$k$ selector; black paths denote the surrounding MLA computation.

As the diagram shows, the indexer runs alongside MLA inside every transformer block. In the released implementation, the indexer query branches off the same normalized low-rank query latent ($q_t^{\text{lora}} \in \mathbb R^{1536}$) used by main attention, whereas the historical indexer key $k_s^I \in \mathbb R^{128}$ is projected directly from the layer's input hidden state $h_s$ and stored in its own lightweight cache.

The lightning indexer scoring equation

For a query token at position $t$ and a candidate historical token at position $s \le t$, the lightning indexer computes a scalar relevance score $I_{t,s}$:

Each variable plays a specific role in keeping the scan fast:

Notice that $k_s^I$ has no head index $j$: just as MLA in MQA mode shares one latent vector across all query heads, the indexer shares a single 128-dimensional key vector $k_s^I$ across all 64 indexer query heads. Replacing a softmax with $\operatorname{ReLU}$ avoids expensive row-wise exponentiation and normalization during the scan, and executing the inner products in FP8 keeps memory bandwidth and arithmetic cost small. Because the head weights $w_{t,j}^I$ come from an unconstrained linear projection of $h_t$, individual weights $w_{t,j}^I$ can be positive or negative even though $\operatorname{ReLU}(q_{t,j}^I \cdot k_s^I) \ge 0$.

How head weights shape candidate selection

To see how the query-dependent head weights $w_{t,j}^I$ steer selection, consider a two-head toy slice with weights $w_t^I = (0.75, 0.25)$ scoring four historical tokens:

With $w_t^I = (0.75, 0.25)$, a top-2 selector picks Position A and either B or C. If the next query token shifts its head weights to $(0.20, 0.80)$, Position B jumps to $6.60$ and Position D rises to $4.80$, completely changing which historical tokens get loaded into MLA.

Once the selector passes those top-$k$ indices to MLA, main attention computes its own softmax weights from scratch using the full 576-dimensional MLA latents; the indexer score $I_{t,s}$ only determines membership in the active set $\mathcal S_t$, not the final attention weight inside MLA.

Shared token-level top-$k$ selection

During sparse training and inference, each query $t$ selects at most $k = 2{,}048$ historical positions:

If $t \le 2{,}048$, the prefix has no more than $k$ tokens and DSA reads the entire history identically to dense MLA. Once $t > 2{,}048$, $\mathcal S_t$ contains the 2,048 indices with the highest $I_{t,s}$ scores.

Three structural properties of $\mathcal S_t$ stand out:

1. Shared across main query heads: At a given layer and token $t$, all 128 MLA heads attend over the same 2,048 selected positions $\mathcal S_t$ (though each head computes its own independent softmax distribution over those 2,048 entries). Sharing $\mathcal S_t$ across heads is what allows the sparse MQA kernel to load each selected latent $c_s^{KV}$ once and reuse it 128 times.
2. Token-level granularity: Selection picks individual token indices rather than coarse 64-token or 128-token page blocks, avoiding wasted budget on irrelevant tokens that happen to sit next to a single critical keyword.
3. No hardcoded local window: Unlike sliding-window + sink-token architectures, the V3.2 DSA prototype does not reserve fixed slots for initial sink tokens or a mandatory local window; any recency bias or anchor-token retention is learned directly by the indexer (which has its own RoPE dimensions).

Enforcing causality across both stages

To prevent information leakage from future tokens $s > t$, causal masking must hold in both the indexer and main attention. During prefill, model.py adds an upper-triangular $-\infty$ mask to $I_{t,s}$ before calling torch.topk.

A subtle boundary case arises during prefill for early tokens where $t < 2{,}048$: when a fixed-width topk(2048) kernel runs on row $t$ of a rectangular score matrix, the remaining $2{,}048 - t$ slots in the returned index tensor will be filled with $-\infty$-scored future indices $s > t$. Consequently, the main MLA stage must still apply its own causal mask (or filter out indices $s > t$) so those padding slots receive zero probability in the softmax. During single-token decoding, the cache slice only contains positions $s \le t$, so future tokens are absent by construction.

Hadamard rotation and FP8 quantization in model.py

Inspecting the Indexer class in inference/model.py reveals several engineering refinements beyond Equation (1) of the paper:

• Partial RoPE: Each 128-dimensional indexer query and key splits into a 64-dimensional RoPE half (qk_rope_head_dim = 64) and a 64-dimensional non-RoPE half, concatenated back to width 128.
• Fast Hadamard Transform (FHT): Before quantizing $q_{t,j}^I$ and $k_s^I$ to FP8, rotate_activation multiplies both vectors by a normalized Walsh-Hadamard matrix $H$ ($H^\top H = I$) and scales by $1/\sqrt{128}$. Because $H$ is orthogonal, $(Hq)^\top(Hk) = q^\top k$ in exact arithmetic, while spreading activation outliers evenly across coordinates so per-tensor/per-token FP8 quantization loses much less precision.
• Fused weight scaling: The head weights $w_{t,j}^I$ are multiplied by $d_{\text{head}}^{-0.5} = 128^{-0.5}$, the softmax scale factor, and the per-token FP8 query quantization scale before being passed into the fp8_index kernel, folding three scalar multiplications into one.

The retrieval bottleneck

Why not simply use main attention's own query-key dot products to pick the top 2,048 entries? Because evaluating 128 heads of 576-dimensional dot products across all $s \le t$ is already half the cost of dense MLA. The indexer cuts that scanning cost by reducing the head count from 128 to 64, shrinking the head dimension from 576 to 128, sharing a single 128-byte FP8 key across heads, and using $\operatorname{ReLU}$ instead of softmax.

The trade-off is a hard retrieval bottleneck: if the lightning indexer fails to rank a critical historical token inside the top 2,048 at layer $\ell$, main attention at layer $\ell$ gets zero signal from that token and cannot send gradients back through the discrete topk operator to fix the miss. Training the indexer to track main attention accurately before switching on sparsity is therefore essential—and is the subject of Chapter 6.

Sources: Report §2.1 (sections/01-method.tex); pinned inference/model.py (Indexer class) in the official repository. Continue with continued training.

6. Converting a Dense Model to Sparse Attention

DeepSeek-V3.2 does not train a 671B sparse-attention model from random initialization. Instead, it starts from the dense DeepSeek-V3.1-Terminus base checkpoint—already extended to a 128K context window—and converts it to DeepSeek Sparse Attention (DSA) through a two-stage continued pre-training curriculum totaling roughly 945.8B tokens.

Retrofitting sparse selection onto a pre-trained dense model poses an obvious bootstrap problem: if you turn on top-2,048 truncation while the lightning indexer weights are still random, every attention layer suddenly loses access to 98% of a 128K context window, corrupting hidden states and destabilizing gradients across the backbone. DeepSeek avoids this shock by warming up the indexer against the frozen dense model before switching on sparse selection.

Stage 1: Dense warm-up (freezing the backbone to train the selector)

In the dense warm-up stage, all inherited backbone parameters remain frozen and main attention continues to run densely over the full causal prefix. Only the newly initialized lightning indexer parameters are updated.

To give the indexer a dense supervision target without hand-labeled retrieval annotations, the authors extract the actual softmax attention distributions $a_{t,s,i}$ computed by the 128 main MLA heads at query $t$, sum them across all heads $i$, and L1-normalize the result across historical positions $s \le t$:

Because each head's causal attention distribution already sums to $1$ over $s \le t$, normalizing their sum simply averages the 128 head distributions into a single consensus distribution $p_{t,:}$ over the prefix. That consensus target matches the structural constraint of DSA, which picks one shared set of 2,048 token positions for all 128 MLA heads.

The indexer scores $I_{t,:}$ are passed through a softmax over $s \le t$, and the indexer is trained to minimize the forward Kullback-Leibler (KL) divergence from the teacher distribution $p_{t,:}$:

Using the forward KL divergence $D_{\mathrm{KL}}(p \| q) = \sum_s p_s \log(p_s / q_s)$ is a deliberate choice: forward KL is mean-seeking (zero-avoiding), meaning it heavily penalizes the student distribution $q = \operatorname{Softmax}(I_{t,:})$ whenever $q_{t,s}$ assigns near-zero probability to a position $s$ where any group of main attention heads placed substantial mass $p_{t,s}$.

For example, if four historical positions receive consensus attention mass $p = (0.60, 0.25, 0.10, 0.05)$ and an untrained indexer initially assigns $q = (0.05, 0.10, 0.25, 0.60)$, a top-2 selector would drop the token carrying 60% of the teacher's attention weight. Minimizing $D_{\mathrm{KL}}(p \| q)$ rapidly pulls the indexer's top logits into alignment with the high-mass positions of the frozen dense model.

Warm-up compute budget

The dense warm-up stage is short and uses a relatively high learning rate of $10^{-3}$ (safe because the pre-trained backbone is frozen). Training runs for 1,000 steps with a batch size of 16 sequences of 128K tokens, which with $128\text{K} = 131{,}072$ tokens gives:

Stage 2: Sparse training (co-adapting the backbone and selector)

Once the indexer can reliably identify the tokens that dense MLA attends to, Stage 2 turns on top-$k$ sparse selection ($k = 2{,}048$) and unfreezes all backbone parameters so the model can adapt its representations to sparse reading.

During this stage, main MLA only computes attention over the selected index set $\mathcal S_t = \operatorname{TopK}_{s \le t}(I_{t,s})$. Consequently, the teacher distribution $p_{t,\mathcal S_t}$ is only available on the selected subset $\mathcal S_t$, and the indexer's KL loss is restricted to that same active support:

Here, the main attention weights $a_{t,s,i}$ are already softmax-normalized over $s \in \mathcal S_t$, and the indexer softmax is likewise taken over the 2,048 logits in $\mathcal S_t$. As the un-frozen backbone adjusts its attention heads to work within a 2,048-token budget, this within-set KL loss keeps the indexer's relative ranking calibrated to the evolving main attention heads.

Detached gradients: why $\mathcal L_{\mathrm{LM}}$ and $\mathcal L^I$ are decoupled

A crucial implementation detail in Stage 2 is that the indexer inputs are detached from the computation graph (stop_gradient), separating the backward pass into two independent optimization paths:

1. Backbone update: The standard next-token cross-entropy loss $\mathcal L_{\mathrm{LM}}$ backpropagates through the sparse MLA layers, MoE routers, and experts, updating the main model weights without sending any gradients into the indexer.
2. Indexer update: The KL alignment loss $\mathcal L^I$ backpropagates only into the lightning indexer parameters; because its hidden-state and query-latent inputs are detached, $\mathcal L^I$ never perturbs the backbone's residual stream or query projections to make the indexer's job artificially easy.

Because the discrete $\operatorname{TopK}$ operator has zero gradient almost everywhere, DeepSeek does not attempt to differentiate $\mathcal L_{\mathrm{LM}}$ through token selection via straight-through estimators or Gumbel-Softmax relaxations. Instead, language modeling adapts the reader to whatever 2,048 tokens are selected, while KL distillation keeps the selector tracking the reader's preferences.

Sparse adaptation compute budget

Stage 2 drops the learning rate to $7.3 \times 10^{-6}$ and runs for 15,000 steps at a batch size of 480 sequences of 128K tokens:

At nearly a trillion tokens—roughly 450 times the warm-up budget—Stage 2 is a major continued pre-training run in its own right. Both stages sample from the exact same 128K long-context extension corpus used for DeepSeek-V3.1-Terminus, ensuring that comparisons between Terminus and the resulting DeepSeek-V3.2-Exp checkpoint isolate the transition from dense MLA to DSA rather than a shift in data mixture.

Trade-offs of attention-matching supervision

Using summed attention weights as the retrieval target is pragmatic, though it comes with two theoretical approximations worth noting:

• Attention weight vs. causal importance: A high softmax weight $a_{t,s,i}$ does not always mean token $s$ is critical to the final output (for instance, attention sinks often absorb large softmax weights associated with near-zero value norms), and conversely, a head with a small weight $a_{t,s,i}$ multiplied by a large value vector $v_{s,i}$ might carry a crucial signal that gets diluted when averaging $a_{t,s,i}$ across all 128 heads.
• No out-of-set supervision in Stage 2: Because Stage 2 only evaluates $\mathcal L^I$ on $s \in \mathcal S_t$, a token $s \notin \mathcal S_t$ that falls just below the 2,048 cutoff receives no direct dense-teacher comparison during Stage 2; keeping those borderline tokens ranked properly relies on the Stage 1 warm-up initialization and the shared representations of the 2,048 tokens that do make the cut.

Even with those approximations, the benchmark comparison between V3.1-Terminus and V3.2-Exp (examined in Chapter 19) shows that two-stage distillation plus 943.7B tokens of sparse adaptation preserves short- and long-context quality across standard evaluations, setting the stage for the systems cost analysis in Chapter 7.

Sources: Report §2.1.1 and §2.2 (sections/01-method.tex). Continue with computation, memory, and inference cost.

7. Computation, Memory, and Inference Cost

DeepSeek Sparse Attention (DSA) makes the heavy MLA reader sparse, but it does not eliminate the lightning indexer's prefix scan, shrink the total KV cache footprint, or reduce the FLOPs spent in MoE feed-forward layers. Building an accurate mental model of V3.2's performance requires separating the pieces that scale with the sparse budget $k = 2{,}048$ from the pieces that still scale with the full sequence length $L$.

Counting query-key pairs in main attention

Over a prompt of length $L$, dense causal attention evaluates all lower-triangular pairs $(t, s)$ with $1 \le s \le t \le L$, totaling $L(L+1)/2$ interactions per layer. Under DSA, each query $t$ attends to at most $k$ historical tokens, capping the total main-attention pairs during prefill at:

When $L \ge k$, the first $k$ tokens attend densely to their short prefixes while the remaining $L - k$ tokens each attend to exactly $k$ entries:

Plugging in the paper's full context window $L = 131{,}072$ and sparse budget $k = 2{,}048$ illustrates the gap between prefill and final-token decode:

• Full-sequence prefill pairs: Dense causal attention evaluates $\frac{131{,}072 \times 131{,}073}{2} \approx 8.59\times 10^9$ pairs per layer, whereas sparse main attention evaluates $131{,}072 \times 2{,}048 - \frac{2{,}048 \times 2{,}047}{2} \approx 2.663\times 10^8$ pairs—a 32.25$\times$ reduction in main-attention pair count across the whole sequence.
• Final-token decode pairs: When decoding a single token at the end of a 131,072-token prefix, dense attention scans all 131,072 cached entries while sparse main attention reads only 2,048 entries—a 64$\times$ reduction in candidate reads for that step.

The prefill ratio ($32.25\times$) is roughly half the final-token decode ratio ($64\times$) because the average prefix length across a prompt of length $L$ is $L/2$, not $L$.

Why full-sequence complexity is still asymptotically $\Theta(L^2)$

Although main attention drops from $\Theta(L^2)$ to $\Theta(Lk)$ during prefill, the lightning indexer must still score every valid candidate pair $(t, s)$ with $s \le t$ to determine which $k$ indices win. Writing out the total prefill cost of a transformer block as a function of $L$ gives three scaling regimes:

Here, $a_I L^2$ captures the quadratic prefix scan inside the lightning indexer, $a_A Lk$ captures the sparse MLA read over $k$ tokens, $bL$ aggregates token-wise linear work (QKV/output projections and the 9 active MoE experts per token), and $C_{\mathrm{topk}}(L)$ represents the top-$k$ selection kernel.

Because $a_I > 0$, the asymptotic complexity as $L \to \infty$ for fixed $k$ remains $\Theta(L^2)$. What makes DSA fast at $L = 128\text{K}$ is the ratio of the leading constants $a_I \ll a_A$:

1. Fewer heads and narrower vectors: The indexer computes 64 heads of width 128 ($64 \times 128 = 8{,}192$ multiply-adds per pair, with No-RoPE/RoPE fused and no value aggregation), whereas main MLA in MQA mode computes 128 heads of 576-dim score inner products plus 128 heads of 512-dim value accumulations ($128 \times (576 + 512) = 139{,}264$ multiply-adds per pair)—more than a $17\times$ reduction in raw arithmetic per pair.
2. FP8 execution and $\operatorname{ReLU}$: Running the indexer dot products in FP8 with a cheap $\operatorname{ReLU}$ activation instead of BF16/FP16 with row-wise softmax synchronization cuts both memory bandwidth and tensor-core cycles further.

During single-token decoding at prefix length $L$, the per-token cost scales linearly with $L$ in the indexer and stays constant in $k$ for main attention:

Thus, decoding a 20,000-token reasoning chain still pays a linear $\alpha_I L$ scan in the indexer at every step, plus the fixed $\alpha_A k$ sparse MLA read and $c_{\mathrm{local}}$ MoE expert execution.

Sparse reads do not shrink the KV cache footprint

A common misconception is that attending to 2,048 tokens means the engine only needs to keep 2,048 tokens in GPU memory. Because token selection is dynamic, a token ignored at step $t$ may be selected at step $t + 50$. The serving engine must therefore retain the entire historical sequence in HBM—both the MLA cache and the new indexer key cache.

In Chapter 4, we calculated that a 131,072-token MLA cache (512 latent + 64 RoPE dimensions) takes ~9.21 GB in BF16 (or ~4.61 GB in FP8). On top of that, DSA must cache one 128-dimensional FP8 indexer key $k_s^I$ plus one 4-byte FP32 quantization scale per token per layer (k_cache and k_scale_cache in model.py):

So DSA actually increases total KV cache storage per sequence by roughly 1 GB at 128K context in exchange for cutting the memory bandwidth read per token during decoding by an order of magnitude.

Kernel engineering vs. the reference model.py script

Achieving wall-clock speedups from token-level sparsity on modern GPUs requires specialized kernels (fp8_index and sparse FlashMLA). Naive sparse gather operations over individual tokens suffer from uncoalesced HBM reads unless each fetched cache line performs enough compute to hide memory latency. Because DSA shares the same 2,048 selected token indices across all 128 MLA query heads in MQA mode, a single 576-element gather from the MLA cache feeds all 128 query heads simultaneously inside shared memory/registers, restoring high arithmetic intensity.

If you benchmark the reference inference/model.py script shipped in the repository, be careful not to confuse it with the production kernels: in model.py, the MLA.forward method computes the full dense $L \times L$ (or $1 \times L$) attention score matrix scores first and then overwrites unselected indices with -inf via index_Mask. That reference code is written to verify mathematical equivalence and checkpoint loading; actual FLOP and bandwidth savings require replacing the dense matmul + mask with a fused sparse gather-and-attend kernel such as TileLang/FlashMLA.

Production H800 cost curves

To show how these systems trade-offs play out on real hardware, the authors benchmark end-to-end serving throughput for DeepSeek-V3.1-Terminus and DeepSeek-V3.2 on a cluster of NVIDIA H800 GPUs, converting throughput into dollar cost per million tokens at a reference rental rate of US$2 per H800 GPU-hour:

Reproduced from Report Figure 2 (figures/cost_prefilling.pdf and figures/cost_decoding.pdf). Vertical axes show estimated serving cost (USD per 1M tokens) as a function of token position inside a 128K context window, priced at $2/GPU-hour on H800 hardware.

Several practical behaviors stand out in Figure 2:

• Short-sequence prefill crossover: At very short sequence lengths, sparse indexing and gather overhead can outweigh dense matrix efficiency, so the authors use a custom masked MHA kernel to simulate DSA on short prompts before switching to sparse MQA as sequence length grows.
• Marginal vs. cumulative cost: The horizontal axis plots the marginal cost at a specific token position (32K, 64K, 96K, 128K), not the average cost across a full request. Estimating the total cost of prefilling a 128K prompt or decoding a 32K trajectory requires integrating the area under the curve from the starting position to the final position.
• Amdahl's law at the model level: Even at 128K context, where main attention reads 64$\times$ fewer tokens during decoding, total per-token decoding cost drops by roughly 3–4$\times$ rather than 64$\times$. By Amdahl's law,

the non-attention portion of the forward pass ($1 - f$)—communicating tokens via all-to-all dispatch/combine, executing 9 SwiGLU experts per layer, running MLA projections, and scanning the FP8 indexer cache—places a hard floor on per-token cost.

That per-token savings is what makes the next part of the paper feasible: scaling post-training reinforcement learning to long-horizon reasoning and multi-turn agent trajectories.

Sources: Report §2.3 (sections/01-method.tex); Report Figure 2 (figures/cost_prefilling.pdf, figures/cost_decoding.pdf); pinned inference/model.py (Indexer and MLA classes). Continue with post-training: specialists, distillation, and mixed RL.

8. Specialists, Distillation, and Mixed Learning

Converting DeepSeek-V3.1-Terminus into the sparse-attention base checkpoint (DeepSeek-V3.2-Exp base) only solves the pre-training half of the problem. Turning that base checkpoint into a general assistant that can prove theorems, debug repositories, orchestrate search tools, and hold normal conversations requires a three-stage post-training pipeline: training dedicated domain specialists via large-scale RL, distilling their trajectories into a unified student checkpoint, and running a final mixed-domain GRPO stage.

Throughout all three post-training stages, the model retains the exact same DeepSeek Sparse Attention (DSA) architecture introduced during continued pre-training.

Stage 1: Training six domain specialists

Instead of jumping straight from the base model into a single multi-task RL run, the authors first branch off six specialized models, each initialized from the same pre-trained V3.2 base checkpoint and trained with large-scale reinforcement learning on a single domain:

1. Mathematics
2. Programming
3. General logical reasoning
4. General agent tasks
5. Agentic coding
6. Agentic search

Alongside these six technical domains, the pipeline maintains specialized teachers for writing and open-ended question answering.

Training separate domain specialists first solves a practical reward-engineering and curriculum problem. A math or competitive-programming specialist benefits from binary outcome verifiers, long rollout limits, and near-zero KL penalties, whereas an agentic search or general chat model requires rubric-based reward models, formatting checks, and stricter style regularization. Decoupling the domains during the initial RL push lets each specialist saturate its domain benchmark without cross-domain gradient interference or reward-scale tuning headaches.

Stage 2: Distilling Thinking and Non-Thinking trajectories

Because deploying six separate 671B checkpoints behind a routing classifier is operationally unwieldy, the specialist models are used as data generators to train a single unified checkpoint. Each specialist generates both thinking mode trajectories (with explicit <think>...</think> scratchpad blocks and interleaved reasoning across tool turns) and non-thinking mode trajectories (direct answers and immediate tool calls).

Training a student policy $\pi_\theta$ on filtered specialist trajectories $(q, o) \in \mathcal D_{\text{teachers}}$ via standard supervised cross-entropy minimizes:

(While the paper describes generating domain-specific data from the specialists to train the merged checkpoint, it does not publish the exact rejection-sampling filters, dataset sizes, or whether token-level logit distillation supplements sequence-level SFT.)

According to the report, the unified checkpoint resulting from specialist distillation lands only slightly below the individual domain specialists on their respective benchmarks—and the subsequent mixed RL stage closes that remaining gap completely.

Supporting both thinking and non-thinking modes inside the same weights gives downstream applications a clean latency-versus-accuracy knob. Both modes execute the exact same 61-layer sparse MoE transformer; the difference is purely behavioral, triggered by the chat template (covered in Chapter 13). On straightforward queries or latency-sensitive agent loops, non-thinking mode skips the multi-thousand-token scratchpad while retaining strong tool-calling accuracy (for instance, scoring 67.6% vs. 73.1% on SWE-bench Verified, examined in Chapter 20).

Stage 3: Unified mixed-domain GRPO

Imitation learning on teacher trajectories has a well-known limitation: once the student model makes a slight mistake at test time, it enters states the teacher never visited. To make the unified model robust on its own rollout distribution—and to erase the small gap left by distillation—DeepSeek-V3.2 runs a consolidated mixed RL stage using Group Relative Policy Optimization (GRPO) that blends reasoning, agent, and human-alignment prompts into a single training loop.

Sequential multi-stage RL (for example, running math RL first, then coding agent RL, then chat RL) notoriously suffers from catastrophic forgetting: optimizing for concise chat responses degrades long-chain math reasoning, while aggressive math exploration degrades instruction formatting. By mixing all domains in every update step, the policy must satisfy all reward signals simultaneously:

• Reasoning and agent tasks: Scored by rule-based outcome verifiers, explicit length penalties, and language-consistency rewards (preventing unwanted mid-sentence language switching).
• General alignment and writing tasks: Scored by a generative reward model that grades responses against prompt-specific rubrics.

Scaling RL compute and branching DeepSeek-V3.2-Speciale

Across specialist training and unified mixed RL, the post-training RL compute budget exceeds 10% of the total pre-training cost. In an agentic RL loop, compute goes not just to forward and backward passes on the 671B policy, but also to autoregressive rollout decoding over 64K–128K contexts, container execution for coding environments, search API queries, and generative reward-model grading.

From this pipeline, the authors produce two distinct final checkpoints:

• DeepSeek-V3.2 (Main): Trained with the full mixed-domain RL recipe (reasoning + agents + general alignment) and calibrated length penalties so token consumption stays practical for everyday production serving.
• DeepSeek-V3.2-Speciale: Branched specifically to probe the upper bound of long-horizon reasoning. Speciale trains exclusively on reasoning data during its RL stage, relaxes the length penalty so the model can allocate tens of thousands of thinking tokens to hard problems, and incorporates the proof dataset and self-verification reward pipeline from DeepSeekMath-V2. Because it skips agent and tool-use RL, Speciale does not support tool calling in the released checkpoint.

Why doesn't cranking RL compute above 10% of pre-training cause the 671B MoE policy to collapse? As the authors detail in Section 3.1 of the report, naive GRPO breaks down at this scale unless you fix three subtle mismatches between rollout generation and gradient updates—the focus of Chapters 9, 10, and 11.

Sources: Report §3 opening and §4.1 (sections/02-post.tex). Continue with GRPO from first principles.

9. GRPO from First Principles

Group Relative Policy Optimization (GRPO) eliminates the separate value-network critic of Proximal Policy Optimization (PPO) by sampling a group of $G$ candidate outputs for each prompt, scoring them with a verifier or reward model, and using the group's average reward as the baseline. For a 671B-parameter MoE model, dropping the critic cuts memory and compute overhead nearly in half while avoiding value-estimation errors over 64K-token reasoning chains.

The three policies in a GRPO update

Every GRPO step involves three distinct probability distributions over next tokens:

Why distinguish $\pi_{\mathrm{old}}$ from $\pi_\theta$? First, to amortize the cost of generating long reasoning and agent trajectories, a single batch of rollouts from $\pi_{\mathrm{old}}$ is typically split into multiple mini-batches for several gradient updates, so $\pi_\theta$ drifts away from $\pi_{\mathrm{old}}$ after the first step. Second, even before a single weight update occurs, the high-throughput inference engine running $\pi_{\mathrm{old}}$ (using FP8 kernels and custom paging) and the distributed training engine running $\pi_\theta$ produce slightly different token probabilities due to floating-point and kernel differences.

Group mean-centered advantages

Given a prompt $q$, the rollout worker samples a group of $G$ outputs $\{o_1, \dots, o_G\}$ from $\pi_{\mathrm{old}}(\cdot \mid q)$ and evaluates their scalar rewards $\{R_1, \dots, R_G\}$. In DeepSeek-V3.2, the advantage assigned to token $t$ of output $o_i$ is the group mean-centered reward:

Notice two specific details in Equation (5) of the report:

1. Sequence-level credit assignment: Every token $t \in \{1, \dots, |o_i|\}$ generated in trajectory $o_i$ receives the exact same outcome advantage $\widehat A_{i,t} = \widehat A_i$.
2. No standard-deviation division: Unlike earlier GRPO formulations that divide by $\operatorname{std}(R_1, \dots, R_G)$, DeepSeek-V3.2's displayed equation subtracts the group mean without dividing by the group standard deviation. Avoiding division by $\operatorname{std}(R)$ prevents exploding advantages when all $G$ samples in a group receive nearly identical rewards (for instance, when only one sample incurs a tiny length or formatting penalty).

Consider a group of $G = 4$ rollouts with binary verification rewards $[1, 0, 0, 1]$. Their mean is $0.5$, yielding advantages $[+0.5, -0.5, -0.5, +0.5]$. If a prompt is so easy that all $G$ rollouts pass ($[1, 1, 1, 1]$) or so hard that all $G$ fail ($[0, 0, 0, 0]$), all outcome advantages become $0$, contributing zero policy-gradient signal from the outcome verifier. Maintaining a task curriculum where prompts sit at intermediate pass rates is therefore critical for sample efficiency.

Token-level importance ratios and the clipped surrogate

Because the tokens $o_{i,t}$ were drawn from $\pi_{\mathrm{old}}$ rather than $\pi_\theta$, the update reweights each token by its conditional likelihood ratio:

Notice that $r_{i,t}(\theta)$ is the ratio of single-token conditional probabilities at prefix $(q, o_{i,<t})$, not the product of ratios across the whole sequence $\prod_{s \le t} r_{i,s}(\theta)$ (which would explode or vanish exponentially over thousands of tokens).

Putting the clipped PPO surrogate, per-response length averaging, and token-level KL penalty together gives the complete GRPO objective maximized by DeepSeek-V3.2:

How length normalization and clipping behave

Two structural terms inside $\mathcal J_{\mathrm{GRPO}}(\theta)$ govern how long trajectories are weighted and bounded:

• Per-trajectory length normalization ($\frac{1}{|o_i|}$): Dividing the inner sum by $|o_i|$ averages the surrogate across tokens within each trajectory before averaging across the $G$ trajectories. Without $\frac{1}{|o_i|}$, a 16,000-token reasoning rollout would contribute 16 times as many token gradient terms as a 1,000-token rollout simply by being verbose. Even with $\frac{1}{|o_i|}$ normalization, however, the reward $R_i$ itself must still include a length penalty if you want the policy to prefer concise solutions over long ones.
• Sign-sensitive PPO clipping: The $\min(\cdot, \cdot)$ around $\operatorname{clip}(r_{i,t}, 1-\varepsilon, 1+\varepsilon)\widehat A_{i,t}$ acts as a pessimistic lower bound.
  • When $\widehat A_{i,t} > 0$ (a better-than-average trajectory), the objective increases with $r_{i,t}(\theta)$ up to $1 + \varepsilon$, above which the flat clipped branch wins and stops the gradient from pushing $\pi_\theta(o_{i,t})$ any higher on that stale sample.
  • When $\widehat A_{i,t} < 0$ (a worse-than-average trajectory), multiplying by a negative number flips the inequality: the objective increases as $r_{i,t}(\theta)$ drops toward $1 - \varepsilon$, below which the flat clipped branch $(1 - \varepsilon)\widehat A_{i,t}$ is more negative than $r_{i,t}\widehat A_{i,t}$ and cuts off further downward pressure.

Credit assignment and action masking in multi-turn trajectories

Assigning a single sequence-level advantage $\widehat A_i$ to every token $t \in \{1, \dots, |o_i|\}$ means GRPO relies on statistical averaging across batches to separate productive reasoning steps from dead-end detours inside a winning trajectory. If a 10,000-token proof meanders for 3,000 tokens before finding the right lemma and succeeding, all 10,000 tokens receive a positive update on that step; only across many groups—especially when paired with a length penalty that gives higher reward $R_i$ to shorter proofs—do redundant detours get washed out.

In multi-turn agent trajectories, the transcript $o_i$ alternates between model-generated tokens (thoughts and tool calls) and environment observations (tool outputs). Because environment observations are injected by the runtime rather than sampled from $\pi_{\mathrm{old}}$, the training loop masks environment tokens out of the inner sum over $t$, conditioning on them in the prefix $(q, o_{i,<t})$ while computing gradients only on tokens emitted by the model.

Finally, look closely at the two places where $\pi_{\mathrm{old}}$, $\pi_\theta$, and $\pi_{\mathrm{ref}}$ interact in $\mathcal J_{\mathrm{GRPO}}(\theta)$: the KL penalty $D_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{ref}})$ and the importance ratio $r_{i,t}(\theta) = \pi_\theta / \pi_{\mathrm{old}}$. In standard RL codebases, subtle bugs in both terms routinely destabilize large MoE runs. Chapters 10 and 11 examine how DeepSeek-V3.2 fixes them.

Sources: Report §3.1 (sections/02-post.tex, Equations 4–5). Continue with the corrected KL estimator.

10. The Corrected KL Estimator

To keep the trainable policy $\pi_\theta$ from drifting too far from the reference checkpoint $\pi_{\mathrm{ref}}$ during GRPO, the objective subtracts a KL penalty $\beta D_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{ref}})$ at every generated token position $t$. Rather than materializing full 129,280-vocabulary softmax distributions for both $\pi_\theta$ and $\pi_{\mathrm{ref}}$ at every token to compute the exact sum, RL implementations typically estimate the KL divergence from the single sampled rollout token $o_{i,t}$ using Schulman's nonnegative K3 estimator.

There is a catch: the K3 estimator is only an unbiased estimator of $D_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{ref}})$ when the token is sampled from $\pi_\theta$. In off-policy GRPO, however, the rollout token $o_{i,t}$ was sampled from $\pi_{\mathrm{old}}$, not $\pi_\theta$. DeepSeek-V3.2 corrects this mismatch by multiplying the K3 estimator by the importance ratio $\pi_\theta / \pi_{\mathrm{old}}$.

Why the standard K3 estimator works on-policy

Fix a single prefix $(q, o_{i,<t})$ and abbreviate the next-token probability distributions over the vocabulary $v \in \mathcal V$ as:

Assume all three distributions have full support on $\mathcal V$ (so $\sum_{v \in \mathcal V} p(v) = \sum_{v \in \mathcal V} q(v) = \sum_{v \in \mathcal V} r(v) = 1$). We want to estimate the forward KL divergence from the reference $r$ to the current policy $q$:

The standard single-sample K3 function is:

Because $x - \log x - 1 \ge 0$ for all $x > 0$ (with equality if and only if $x = 1$), $f(v)$ is guaranteed to be nonnegative for every token $v$, reducing variance compared to the naive log-ratio $\log(q(v)/r(v))$. If we draw $v \sim q$, its expectation is:

What goes wrong when sampling from $\pi_{\mathrm{old}}$

During an off-policy GRPO step, the token $v = o_{i,t}$ was actually drawn from $p = \pi_{\mathrm{old}}$, not $q = \pi_\theta$. Taking the expectation of the uncorrected K3 term $f(v)$ under $v \sim p$ yields:

To see the numerical bias on a two-token vocabulary, take:

The true KL divergence between $q$ and $r$ is:

Evaluating the uncorrected K3 function $f(v)$ on the two tokens gives $f(1) \approx 0.31629$ and $f(2) \approx 0.13000$. Because the rollout policy $p = (0.8, 0.2)$ samples token 1 four times as often as token 2, the uncorrected expectation under $p$ is $0.8(0.31629) + 0.2(0.13000) \approx \mathbf{0.27903}$, overestimating the true KL divergence by 25%.

Even worse than the value bias is the gradient pathology when $q(v) \ll r(v)$. Differentiating the uncorrected K3 term $f(v)$ with respect to $\theta$ gives:

When the current policy $\pi_\theta$ assigns very small probability $q(v)$ to a token $v$ that was sampled by $\pi_{\mathrm{old}}$ and favored by $\pi_{\mathrm{ref}}$, the coefficient $r(v)/q(v)$ explodes. In uncorrected K3 implementations, those rare off-policy tokens hit $\pi_\theta$ with massive, unclipped gradient spikes that push probability mass violently back toward $\pi_{\mathrm{ref}}$ and destabilize training.

DeepSeek-V3.2's importance-weighted K3 estimator

Equation (6) of the report fixes both the expectation mismatch and the $r(v)/q(v)$ gradient spike by multiplying the K3 term by the importance weight $\frac{q(v)}{p(v)} = \frac{\pi_\theta(o_{i,t}\mid q,o_{i,<t})}{\pi_{\mathrm{old}}(o_{i,t}\mid q,o_{i,<t})}$: