Introduction

DeepSeek-V4 is a family of Mixture-of-Experts language models designed to make million-token contexts practical to train and serve. Rather than relying on a single kernel optimization or cache compression trick, the architecture pairs sequence-compressed attention with multi-stream residual connections, matrix-orthogonalized optimization, low-precision expert routing, and multi-teacher post-training distillation.

This book walks through the DeepSeek-V4 technical report (DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence, preview snapshot), tracing how its architectural mechanisms work, how their compute and memory costs scale, how the training and serving systems are built, and where the paper leaves implementation details unspecified.

How to read the book

Chapters 1–3 introduce the long-context bottleneck, establish core Transformer and Mixture-of-Experts terminology, and map out the V4-Flash and V4-Pro model configurations. Chapters 4–10 examine the architecture in detail: MoE routing, multi-token prediction, Compressed Sparse Attention (CSA), Heavily Compressed Attention (HCA), long-context cost scaling, manifold-constrained hyper-connections (mHC), and the Muon optimizer. Chapters 11–17 cover pre-training stability, distributed kernels, heterogeneous KV cache serving, specialist reinforcement learning, on-policy distillation, FP4 quantization-aware training, and agent rollout infrastructure. Chapters 18–20 evaluate the benchmark and human-study results alongside practical deployment trade-offs, while Chapters 21–22 provide an end-to-end trace of a single request and a consolidated reference of notation, parameters, and source mappings.

Each chapter is self-contained enough to be read directly when looking up a specific subsystem, with worked numerical examples alongside the equations.

Contents

Reading paths

Depending on what you want to extract from the report, several direct routes through the chapters work well:

• Architecture: Read Chapters 2–10, followed by the end-to-end trace in Chapter 21.
• Training and optimization: Read Chapters 3, 9–12, and 14–16.
• Inference and serving systems: Read Chapters 5–8, 12–13, and 16–17.
• Evaluation and model selection: Read Chapters 1, 3, and 18–20.

Conventions and units

While the report details the core equations and systems architecture, it omits parts of the training recipe—such as exact data mixture weights, several stage-specific compute budgets, and architectural conventions inherited from DeepSeek-V3—and does not include isolated ablations for every component in the combined model. Where a mechanism relies on earlier DeepSeek papers or open-source configuration files, the text notes that explicitly.

Throughout the book, numerical examples use decimal units unless stated otherwise: 1M means $1{,}000{,}000$, and 1 GB means $10^9$ bytes. The released V4-Pro configuration specifies a maximum position count of $1{,}048{,}576$ ($2^{20}$), so cost calculations evaluated at $1{,}000{,}000$ tokens serve as clean decimal approximations to the architectural limit; Chapter 22 documents both figures. Source citations at the end of each chapter reference the section, table, and line range in the local main.tex snapshot.

1. Why a Million Tokens Changes the Problem

As a sequence grows from a few thousand tokens to a million, reading past context stops being a minor overhead and becomes the dominant cost of language model inference. Multi-turn software engineering agents, deep-research workflows, and multi-document analysis routinely accumulate hundreds of thousands of tokens of file contents, test logs, search results, and intermediate reasoning. DeepSeek-V4 addresses this regime by redesigning how the Transformer stores and attends to its history: compressing older token blocks into learned summary entries, selecting only the highest-scoring entries for fine-grained attention, and keeping a short uncompressed window for immediate local context.

What a million-token context permits

A model's context window bounds the total sequence it can condition on in a single pass—not just the user's immediate prompt, but system instructions, prior conversation turns, retrieved documents, tool outputs, and the model's own generated reasoning and actions.

In an interactive coding session, an agent inspects repository files, runs test suites, reads stack traces, edits patches, and revisits earlier design decisions. Every tool invocation appends new text to the trajectory. A million-token window lets the runtime keep that working history intact without aggressively truncating logs or summarizing away exact line numbers and error messages. Similarly, in document analysis, fitting an entire collection of financial filings or technical specifications into context allows cross-document synthesis while keeping the verbatim source text available for follow-up verification.

Fitting a million tokens into memory does not guarantee that the model will use them well; longer contexts also introduce more distractors and competing passages. Building a usable million-token system requires solving both the systems problem of fitting and scanning the history and the modeling problem of retrieving and reasoning over relevant details buried inside it.

The two phases of inference

Transformer serving splits into two distinct execution regimes:

• Prefill processes the input prompt in parallel, computing hidden states across all layers and populating the key–value (KV) cache before emitting the first output token. On long prompts, prefill compute and prefix-cache hit rates determine time to first token (TTFT).
• Decode generates output tokens autoregressively, one step at a time. Each step computes the new token's representations, attends over the cached history, and appends the new state to the cache.

Architectural changes can affect prefill and decode differently. When the DeepSeek-V4 report quotes headline FLOP reductions at a million-token context, those figures measure single-token inference at a given context length—the marginal cost of processing one token against an already-populated million-token history—rather than the total cost of prefilling all one million tokens from scratch or end-to-end wall-clock latency.

Why ordinary attention becomes expensive

In a standard causal Transformer, token $t$ attends to all preceding positions $1, \dots, t$. Across a full prefill of $N$ tokens, the number of query–key interactions per head scales quadratically:

During decode, each newly generated token scans the $N$ cached prompt tokens plus any previously generated tokens. Generating $T$ output tokens after a prompt of length $N$ requires

past-reading interactions per head before multiplying by head dimension and layer count. For reasoning models that emit tens of thousands of chain-of-thought tokens, every additional reasoning step must scan the entire prompt plus the growing chain of thought, making per-token history cost a direct constraint on test-time compute budgets.

At the same time, storing the KV cache creates a severe memory bottleneck. Because the server caches keys and values at every layer to avoid recomputing the prompt on every decode step, cache capacity grows linearly with $N$. Under concurrent long-context requests, KV cache memory quickly outstrips the memory needed to hold the model weights themselves.

Quadratic attention compute and linear KV cache footprint require different remedies:

Because no single technique eliminates both compute and storage growth, DeepSeek-V4 combines head sharing, sequence compression, and sparse selection.

Three ways to make history cheaper

DeepSeek-V4 reduces the cost of historical context along three axes:

1. Head sharing (Shared KV MQA): All query heads within a layer read from a single shared key–value vector rather than maintaining separate KV heads or separate key and value tensors.
2. Sequence compression: Learned pooling modules condense blocks of consecutive tokens into single memory entries along the sequence dimension, shrinking both the stored cache and the candidate pool.
3. Sparse selection: In Compressed Sparse Attention (CSA), a lightweight indexer scores compressed entries and retrieves only the top-$k$ blocks for the main attention heads to read.

Each reduction introduces its own modeling trade-off:

To prevent compression boundaries from hiding recent tokens, both attention variants also maintain a 128-token uncompressed local sliding window.

Why there are two compressed attention types

DeepSeek-V4 interleaves two distinct compressed attention mechanisms across its layers:

• Compressed Sparse Attention (CSA) compresses at a modest 4:1 ratio ($m = 4$), using overlapping pooling windows of eight tokens with a stride of four so adjacent entries share boundary context. Rather than attending to all $N/4$ entries, an indexer selects at most $k$ entries per query ($k = 512$ in Flash, $k = 1{,}024$ in Pro).
• Heavily Compressed Attention (HCA) compresses much more aggressively at a 128:1 ratio ($m' = 128$). Because a million-token history collapses to just $\approx 7{,}812$ HCA entries, the query attends densely to every eligible compressed entry without needing a sparse selector.

Pairing the two gives the network complementary views of the past: CSA preserves relatively fine-grained resolution for targeted retrieval of specific passages, while HCA maintains a coarse, full-sequence view of the entire document or trajectory without top-$k$ selection blind spots. While the paper does not provide layer-by-layer probing ablations to prove how work divides between CSA and HCA layers, interleaving them avoids forcing a single compression stride to handle both global summary and fine-grained lookup.

What the headline efficiency claims mean

At a one-million-token context length, the paper reports substantial reductions in single-token inference FLOPs and KV cache size relative to DeepSeek-V3.2:

In this comparison, FLOPs are normalized to equivalent FP8 FLOPs, a convention that discounts low-precision (FP4) matrix operations according to their reduced arithmetic cost.

Notably, V4-Pro achieves 27% of V3.2's single-token FLOPs at 1M context despite activating more parameters per token (49B active in V4-Pro versus 37B active in V3.2). At short contexts, where MoE expert weights dominate per-token compute, V4-Pro requires more arithmetic than V3.2. At one million tokens, however, reading historical KV state dominates the budget, so shrinking attention work outweighs the larger feed-forward network. Consequently, these 1M-context ratios cannot be extrapolated down to short 1K- or 4K-token prompts.

Compression and information retention

Packing four or 128 tokens into a single 512-dimensional vector inevitably discards information; a fixed-size vector cannot losslessly encode arbitrary sequences of tokens. The practical question is whether end-to-end training teaches the compressor and indexer to preserve the specific features—negations, numeric figures, entity bindings, function signatures—that downstream tasks require.

In a contract or technical specification, dropping a single qualifier or version number changes the meaning of a clause, and even if the 4:1 CSA compressor preserves that detail, the sparse indexer must still rank that block inside the top $k$ for the main attention heads to see it. Evaluating a compressed-attention model therefore requires testing both needle-in-a-haystack retrieval and multi-document synthesis under heavy distractor load, topics we return to in Chapters 18 and 19.

Beyond the attention layer

Making compressed attention work at the 1.6-trillion-parameter scale requires matching changes across the rest of the training and serving stack:

• Architecture and optimization: Manifold-constrained hyper-connections (mHC) widen the residual stream without widening the attention or MoE layers, while the Muon optimizer applies orthogonalized matrix updates alongside MoE routing and activation-clamping safeguards.
• Systems and serving: Custom MegaMoE and TileLang kernels handle variable-length sparse attention and expert dispatch, paired with a tiered inference cache that manages CSA, HCA, and uncompressed window buffers across GPU memory and host storage.
• Post-training and quantization: Domain-specialized RL teachers are consolidated into a single production student via on-policy distillation, and quantization-aware training adapts both the MoE experts and the CSA indexer for FP4 execution.

Sources: main.tex, Introduction (lines 190–270), attention overview and efficiency comparisons (lines 365–385, 533–545).

Continue with the foundations or return to the contents.

2. The Concepts the Paper Assumes

DeepSeek-V4 builds on several specialized variants of Transformer attention, Mixture-of-Experts routing, and reinforcement learning post-training. Because the paper's notation overloading—such as "dense" attention inside a sparse MoE model, or "experts" referring both to feed-forward sublayers and to post-training teacher models—can easily cause confusion, this chapter establishes the core representations and terminology used throughout the rest of the book.

Tokens and hidden states

A tokenizer segments raw text into a sequence of integer IDs representing words, subwords, punctuation, or byte sequences; depending on language and code density, a million tokens typically corresponds to roughly 700,000–800,000 English words or several hundred thousand lines of source code.

An embedding table maps each token ID to an initial vector of width $d$. For a sequence of $N$ tokens, the input to a Transformer layer forms a matrix

where row $t$ holds the hidden state of token $t$. After passing through all layers, the final hidden state is projected by the output head to a vector of vocabulary logits $z$, which a softmax converts into next-token probabilities:

In autoregressive generation, the server samples or selects the next token from $p(v)$, appends it to the sequence, and runs the next forward pass.

What an attention head computes

At each layer, an attention head projects the current token's hidden state into a query vector $q$, scores it against key vectors $K$ from accessible positions, and computes a weighted sum of value vectors $V$:

where $c$ is the head dimension and the causal mask sets future positions to $-\infty$. Running multiple query heads in parallel allows a layer to match different features simultaneously before projecting the concatenated head outputs back to width $d$.

DeepSeek-V4 alters standard causal attention in two fundamental ways: first, most keys and values represent compressed blocks of multiple tokens rather than single token positions; second, within each compressed stream, a single shared vector serves simultaneously as both the key and the value.

MHA, GQA, MQA, and MLA

Modern decoder architectures differ primarily in how they share or compress key and value projections across query heads:

• Multi-Head Attention (MHA) assigns every query head its own dedicated key and value head, maximizing representational independence at the cost of a large KV cache.
• Grouped-Query Attention (GQA) partitions query heads into groups that share a single key and value head (for instance, 64 query heads sharing 8 KV heads).
• Multi-Query Attention (MQA) shares a single key and value head across all query heads in the layer, keeping query projections distinct while minimizing KV storage.
• Multi-Head Latent Attention (MLA), introduced in DeepSeek-V2 and V3, compresses keys and values into a low-rank joint latent vector along the channel dimension and absorbs the up-projection matrices into the query and output projections during inference.

DeepSeek-V4 retains low-rank latent projections for its queries (compressing hidden state $d$ to $d_c$ before expanding to $n_h$ heads), but replaces MLA's key–value latent expansion with shared key–value MQA: all query heads read directly from a single shared 512-dimensional vector that acts as both key and value. While MQA reduces storage along the head dimension, V4's sequence compressors reduce storage along the token dimension, allowing the two savings to multiply.

Why the KV cache exists

During autoregressive decoding, recomputing keys and values for all preceding tokens at every step would make generation prohibitively slow. Instead, the inference server caches the key and value tensors produced at each layer.

In a conventional Transformer with $L$ layers, $H_{\mathrm{KV}}$ key–value heads, head dimension $c$, and $b$ bytes per element, the raw payload size for $N$ tokens is

where the factor of $2$ accounts for storing separate keys and values. Tying keys and values into a single shared vector ($H_{\mathrm{KV}}=1$, no factor of $2$) and compressing the sequence length from $N$ to $N/m$ dramatically shrinks this payload, though practical serving systems must still allocate memory for quantization scale factors, block-table metadata, padding, and uncompressed local window buffers.

Residual connections and multi-stream state

In a standard pre-norm Transformer, each sublayer (attention or feed-forward) reads the residual stream, applies normalization and its learned transformation $F_l$, and adds the result back:

This additive path lets gradients flow unimpeded across dozens of layers. DeepSeek-V4 replaces the single residual vector per token with Manifold-Constrained Hyper-Connections (mHC), maintaining $n_{\mathrm{hc}}=4$ parallel residual vectors of width $d$ per token. Before each attention or MoE sublayer, learned coefficients collapse the four streams into a single vector of width $d$; after the sublayer runs, learned projections distribute the output back across the four streams while mixing the streams via a doubly stochastic matrix. Because the collapse happens before the sublayer, expanding the residual stream by $4\times$ does not multiply the cost of attention or expert matrix multiplications.

Normalization and rotary position embeddings

DeepSeek-V4 applies Root Mean Square Layer Normalization (RMSNorm) throughout the network, rescaling a vector $x \in \mathbb{R}^d$ by its root-mean-square magnitude without mean centering:

where $g$ is a learned per-coordinate gain vector and $\epsilon$ provides numerical stability. Normalizing queries and compressed KV vectors prior to attention bounds logit growth during training, though the learned gain parameters $g$ still allow the model to adjust effective attention temperatures.

To encode sequence order, the model uses Rotary Position Embeddings (RoPE), which rotate pairs of feature coordinates by angles proportional to their sequence position so that $q^\top k$ depends on relative distance. In V4, RoPE is applied to only half of the coordinates in each query and shared KV vector ("partial RoPE"). Furthermore, because the shared KV vector is also read directly as the value $V$, those rotated coordinates carry position-dependent phase shifts into the attention output; as Chapter 7 details, V4 applies an inverse RoPE rotation to the output to unwind that positional phase.

Disambiguating "dense", "sparse", and "expert"

Several standard terms do double duty across different sections of the V4 report:

• Dense vs. sparse networks: A dense feed-forward layer applies all of its weights to every token, whereas a Mixture-of-Experts (MoE) layer routes each token to a small subset of experts. Both V4-Flash and V4-Pro are sparse MoE models in every Transformer block.
• Dense vs. sparse attention: CSA is sparse attention because each query selects at most $k$ compressed entries. HCA is dense attention over its compressed sequence because each query attends to all $\approx N/128$ compressed entries—even though it sits inside an MoE model and operates on heavily compressed tokens.
• MoE experts vs. domain specialists: Inside a Transformer layer, an "expert" is a small SwiGLU-style feed-forward subnetwork (e.g., 384 routed experts per layer in V4-Pro). In the post-training pipeline (Chapters 14–15), an "expert" or "specialist" refers to a full checkpoint fine-tuned with reinforcement learning on a specific domain such as competitive coding, math, or agentic tool use.

Training stages and policy rollouts

Training DeepSeek-V4 proceeds through five distinct regimes:

1. Pre-training optimizes next-token cross-entropy (plus a multi-token prediction head) over 32T–33T tokens, progressively extending the context window and transitioning CSA from dense warmup to sparse top-$k$ selection.
2. Supervised fine-tuning (SFT) trains specialist checkpoints on curated demonstrations of reasoning formats, tool calls, and domain tasks.
3. Reinforcement learning (RL) refines those specialists using Group Relative Policy Optimization (GRPO), scoring model-generated trajectories against code sandboxes, rule verifiers, and generative reward models.
4. On-policy distillation merges the specialist models into a single general-purpose student. Rather than training on static teacher completions (off-policy data), the student generates its own trajectories (on-policy data) and minimizes reverse KL divergence against the token distributions assigned by the corresponding domain teacher.
5. Quantization-aware training (QAT) continues training with simulated FP4 quantization and straight-through gradient estimators so the MoE experts and CSA indexer tolerate low-precision inference without severe degradation.

FLOPs, bytes, and wall-clock time

Throughout our analysis of V4's architecture, we track three distinct hardware metrics:

• FLOPs measure arithmetic operation count, which governs latency in compute-bound regimes such as large-batch prefill or wide matrix multiplications.
• Bytes transferred measure memory capacity and bandwidth demand, which govern latency in memory-bandwidth-bound regimes such as small-batch autoregressive decoding over 1.6T MoE weights or long KV caches.
• Wall-clock latency reflects end-to-end execution time after accounting for kernel launch overhead, irregular sparse gather patterns, all-to-all expert communication, and pipeline synchronization.

Cutting attention FLOPs does not automatically yield an equal speedup in wall-clock time if the indexer's memory reads or distributed expert routing dominate execution. Chapters 8, 12, and 13 examine how these three metrics interact in practice.

Sources: Architectural primitives follow main.tex, Section 2 (lines 278–593); training and post-training terminology follow Sections 3 and 4 (lines 904–1294).

Continue with the model map or return to the contents.

3. The Model in One View

DeepSeek-V4 is released in two model scales: V4-Flash (284B total parameters, 13B active) and V4-Pro (1.6T total parameters, 49B active). Both models share the same million-token context architecture—combining Compressed Sparse Attention (CSA), Heavily Compressed Attention (HCA), four-stream Manifold-Constrained Hyper-Connections (mHC), and Mixture-of-Experts (MoE) layers in every block—while scaling layer depth, hidden dimension, query head count, expert count, and sparse selection budget.

Architectural configuration

The table below consolidates the structural hyperparameters reported for both models:

Note how the early layers configure attention and MoE routing independently:

• Attention schedule: In V4-Flash, the first two layers use pure local sliding-window attention (128 tokens); in V4-Pro, the first two layers use HCA. All remaining layers interleave CSA and HCA.
• MoE routing schedule: In both models, the first three Transformer blocks use MoE layers with deterministic token-ID hash routing, while layers 4 and above use learned top-6 routing.

Inside a single Transformer block

Every Transformer block in DeepSeek-V4 consists of a compressed attention sublayer followed by an MoE feed-forward sublayer, wrapped in a four-stream mHC residual highway. Whereas DeepSeek-V3 used dense feed-forward networks in its first three layers to avoid early routing instability, V4 uses MoE in every block and stabilizes the first three layers by routing tokens via a static hash of their token IDs.

Across the full network, a token's representation flows through the following pipeline:

token ID
  → embedding lookup (width d)
  → 4-stream mHC residual state (4 × d)
  → L Transformer blocks:
      • mHC collapse (4 × d → d) → CSA or HCA attention → mHC mix & add
      • mHC collapse (4 × d → d) → MoE (1 shared + 6 routed experts) → mHC mix & add
  → final stream reduction & RMSNorm
  → output head → vocabulary logits → next-token distribution

The three sequence lengths inside attention

At a given context length $N$, a V4 forward pass operates over three distinct sequence resolutions simultaneously:

1. Uncompressed token positions ($N$): Used within the 128-token local sliding window and as the input to the sequence compressors.
2. CSA compressed blocks ($\approx N/4$): Produced by 4:1 overlapping pooling. The CSA indexer scans all $\approx N/4$ candidate keys, selects up to $k$ entries ($512$ for Flash, $1{,}024$ for Pro), and passes only those $k$ entries (plus up to 128 local window tokens) to the main query heads.
3. HCA compressed blocks ($\approx N/128$): Produced by 128:1 pooling. At $N = 1{,}000{,}000$, this yields $7{,}812$ entries, all of which are read densely by the HCA query heads alongside the 128-token local window.

Because the CSA indexer must still score all $N/4$ compressed blocks and HCA must read all $N/128$ blocks, attention work per token is not strictly constant with respect to $N$; rather, the linear growth term is scaled down into a lightweight indexer scan and a heavily compressed HCA pass.

Query head scaling and grouped output projections

While both models share a single 512-dimensional KV head per compressed stream, V4-Flash uses $n_h = 64$ query heads and V4-Pro uses $n_h = 128$. Concatenating the 512-dimensional outputs across all query heads produces an intermediate vector of width

in V4-Flash and

in V4-Pro—nearly ten times wider than the hidden dimension $d$. Applying a single dense output projection matrix from $65{,}536$ back to $7{,}168$ would add nearly half a billion parameters per layer. Instead, V4 partitions the query heads into $g$ groups ($g = 8$ in Flash, $g = 16$ in Pro, keeping 8 heads per group in both models) and projects each group of $8 \times 512 = 4{,}096$ channels down to an intermediate width $d_g = 1{,}024$ before mapping to $d$.

Total versus active parameters

In V4-Pro, each token activates 6 out of 384 routed experts per layer ($1.56\%$ of routed experts), while in V4-Flash each token activates 6 out of 256 ($2.34\%$). However, every token also passes through the shared expert, the attention projections, the compressors, and the embedding/output layers, which is why active parameters (49B out of 1.6T, or $\approx 3.1\%$, for Pro; 13B out of 284B, or $\approx 4.6\%$, for Flash) exceed the raw routed-expert fraction.

For systems engineering, total and active parameter counts govern different bottlenecks:

• Single-token compute scales with the active parameters (13B or 49B).
• Cluster memory footprint scales with total parameters (284B or 1.6T), because all experts must stay resident in GPU memory. Furthermore, across a large serving batch, different tokens route to different subsets of experts, quickly touching nearly all 256 or 384 experts per layer during a single batch step.

What changes from DeepSeek-V3 and V3.2

DeepSeek-V4 builds directly on the DeepSeekMoE architecture, retaining auxiliary-loss-free load balancing (bias adjustment plus a small sequence-level balance penalty) and a single-step Multi-Token Prediction (MTP) training objective. At the same time, it introduces several major departures from V3 and V3.2:

• Attention: Replaces MLA's per-token latent KV cache (and V3.2's un-compressed DeepSeek Sparse Attention) with sequence-compressed shared-KV MQA, split into 4:1 sparse CSA and 128:1 dense HCA.
• Residual stream: Adds four-stream Manifold-Constrained Hyper-Connections (mHC) projected onto the Birkhoff polytope via 20 Sinkhorn–Knopp iterations.
• MoE routing: Replaces the sigmoid routing gate with $\sqrt{\operatorname{softplus}(x)}$, drops the node-limited routing constraint used in V3, and replaces the first three dense FFN layers with hash-routed MoE layers.
• Optimizer: Replaces AdamW with the matrix-orthogonalized Muon optimizer for all core body matrices, keeping AdamW only for embeddings, output heads, norms, and small structural projections.

Reasoning modes and evaluation budgets

Rather than shipping separate reasoning and non-reasoning models, a single V4 checkpoint supports three operating modes trained during post-training: Non-think, Think High, and Think Max.

In evaluation tables, labels such as V4-Pro-Max or V4-Pro-High denote the same 1.6T-parameter V4-Pro weights run under different system instructions and test-time token budgets:

Although the architecture supports a 1M-token context window ($1{,}048{,}576$ positions), standard reasoning and coding benchmarks are evaluated at these smaller budgets (8K–512K), reserving the full 1M window for long-context evaluations such as MRCR and GraphWalks.

End-to-end training and deployment pipeline

The lifecycle from raw text to a deployed V4 server follows five stages:

1. Corpus curation and pre-training: Train the base model on 32T (Flash) or 33T (Pro) tokens with progressive context extension and delayed sparse-attention activation.
2. Specialist SFT and RL: Fine-tune domain-specific checkpoints using SFT and GRPO across reasoning, coding, math, agentic workflows, and general writing.
3. Multi-teacher on-policy distillation: Roll out student trajectories across all domains and train a unified model to match the token-level distributions of the respective domain specialists via reverse KL minimization.
4. FP4 quantization-aware training: Adapt the unified model so that MoE expert weights and the CSA indexer operate cleanly in FP4 precision.
5. Heterogeneous serving: Deploy the model with MegaMoE and TileLang kernels backed by a multi-tier prefix cache that stores CSA, HCA, and sliding-window states across HBM, host DRAM, and SSDs.

Sources: main.tex, Architecture (lines 278–305), Model Setups (lines 923–957), reasoning modes and evaluation configurations (lines 1063–1110, 1363–1388).

Continue with experts and multi-token prediction or return to the contents.

4. Experts and Multi-Token Prediction

Mixture-of-Experts (MoE) layers decouple a model's total parameter capacity from its per-token arithmetic cost by routing each token through a small fraction of the available feed-forward subnetworks. DeepSeek-V4 builds on the DeepSeekMoE design from V3—combining one shared expert with fine-grained routed experts, auxiliary-loss-free bias balancing, and a single-step Multi-Token Prediction (MTP) objective—while modifying the router's activation function, removing node-limited routing constraints, and replacing the early dense feed-forward layers with deterministic hash-routed MoE blocks.

What an MoE expert computes

Each expert in a V4 MoE layer is a SwiGLU feed-forward network that transforms a single token's hidden vector $h \in \mathbb{R}^d$ via two up-projections ($W_{\mathrm{gate}}, W_{\mathrm{up}}$) and one down-projection ($W_{\mathrm{down}}$):

With hidden dimension $d$ and intermediate width $f$, the three matrices hold $3df$ parameters and require roughly $6df$ FLOPs per token (counting a multiply-accumulate as two operations). In V4-Pro ($d=7{,}168$, $f=3{,}072$), a single expert's three matrices contain

parameters ($\approx 66.1\text{M}$ weights). In V4-Flash ($d=4{,}096$, $f=2{,}048$), each expert holds $3 \times 4{,}096 \times 2{,}048 = 25{,}165{,}824$ parameters ($\approx 25.2\text{M}$ weights).

Every MoE layer pairs one shared expert $E_{\mathrm{shared}}$ (applied unconditionally to every token) with $N_r$ routed experts ($N_r = 256$ in Flash, $N_r = 384$ in Pro), from which the router selects top-$K$ experts ($K = 6$) per token:

where $S(h)$ is the set of 6 selected expert indices and $a_e(h)$ is the normalized gating weight assigned to expert $e$. Dedicating a shared expert to common transformations frees the routed experts to specialize in narrower features without duplicating basic operations across all 256 or 384 subnetworks.

Active parameters versus deployment footprint

Although V4-Pro activates only 49B parameters per token (and Flash activates 13B), all 1.6T (or 284B) parameters must remain accessible in cluster memory. During small-batch decoding, a single token touches only 7 experts per layer (1 shared + 6 routed), but across a large serving batch, different tokens select different combinations of experts, rapidly activating nearly the entire expert pool at every layer.

Conversely, when the batch size per GPU is small, many experts receive only one or two tokens, starving tensor cores of arithmetic intensity and exposing all-to-all network latency. Chapter 12 shows how V4's MegaMoE kernels schedule these irregular workloads across GPU waves.

The square-root softplus routing score

In DeepSeek-V3, token-to-expert affinity scores were computed by passing router logits through a sigmoid function. DeepSeek-V4 replaces the sigmoid with the square root of softplus:

Unlike $\sigma(x)$, which saturates at $1$ for large positive logits, $\sqrt{\operatorname{softplus}(x)}$ grows unboundedly as $\approx \sqrt{x}$ for $x \gg 0$ while decaying exponentially as $\approx \exp(x/2)$ for $x \ll 0$.

At first glance, replacing $\sigma(x)$ with another strictly increasing function might seem unable to change which experts win a top-$K$ selection. However, in DeepSeek's load-balancing scheme, top-$K$ selection does not operate on raw affinities alone—it operates on affinity scores shifted by learned load-balancing biases $b_e$, and the selected experts' outputs are weighted by normalized affinities. Because $\sqrt{\operatorname{softplus}(x)}$ avoids saturating at $1$ on strong matches, the spacing between high-scoring experts and the relative leverage of additive biases $b_e$ behave very differently than under a bounded sigmoid.

Load balancing: bias adjustment and sequence-wise loss

Left unconstrained, learned routers frequently suffer from routing collapse, funneling traffic into a handful of overloaded experts while leaving others undertrained and creating severe hardware stragglers.

DeepSeek-V4 retains V3's auxiliary-loss-free load balancing mechanism paired with a small sequence-wise balance loss:

• Bias-adjusted selection: Each routed expert $e$ maintains a dynamic scalar bias $b_e$, updated at a rate of $0.001$ based on observed token load (decreasing $b_e$ when expert $e$ is overloaded and increasing it when underloaded). The router selects the top $K$ experts using the biased scores $s_e(h) + b_e$, but computes the combination weights $a_e(h)$ from the unbiased affinities $s_e(h)$:

• Sequence-level balance penalty: Even when expert load is balanced across a global training batch, a single long sequence (such as a million-token code file) could still route almost all of its tokens to a few experts, overwhelming the memory and compute of the nodes hosting them. A lightweight sequence-wise auxiliary loss (weighted at $0.0001$) penalizes extreme imbalance within individual sequences.

To see how bias-adjusted selection separates load control from gating weights, consider a top-2 selection over three experts with raw affinities $s = (0.8, 0.7, 0.2)$ and load biases $b = (-0.2, 0.0, 0.6)$. Adding the biases gives $(0.6, 0.7, 0.8)$, so the router selects experts 2 and 3. When weighting their outputs, however, it uses the unbiased scores $0.7$ and $0.2$, assigning weights $0.7 / (0.7 + 0.2) \approx 0.778$ and $0.2 / (0.7 + 0.2) \approx 0.222$.

Hash routing in the first three MoE layers

In DeepSeek-V3, the first three Transformer blocks used dense feed-forward networks because learned MoE routers struggle to balance traffic in the earliest layers where token representations have barely mixed. DeepSeek-V4 replaces those initial dense blocks with MoE layers that route tokens via a predefined deterministic hash function of the input token ID.

Under token-ID hash routing, every occurrence of a given vocabulary token is dispatched to the same six routed experts in layers 1–3 regardless of surrounding context. Because vocabulary token frequencies follow a stationary distribution across the training corpus, static hash buckets provide predictable load balance without needing learned router gradients or dynamic bias adjustments in the bottom three layers.

Removing the node-limited routing constraint

To cap cross-node communication volume, DeepSeek-V3 restricted each token so that its selected experts could span at most $M$ physical nodes. DeepSeek-V4 drops this routing-node constraint entirely, allowing a token's six routed experts to reside on any nodes in the expert-parallel group. While unconstrained routing gives the model full freedom to combine experts, it increases cross-node traffic; Chapter 12 examines how V4's communication-overlapped MegaMoE pipeline absorbs this extra dispatch and combine cost.

Multi-token prediction (MTP)

Alongside the standard next-token cross-entropy loss, DeepSeek-V4 retains V3's sequential Multi-Token Prediction (MTP) objective with depth $1$. At each sequence position $t$, the main Transformer trunk produces hidden state $h_t$ to predict token $x_{t+1}$, while a single additional MTP Transformer block uses $h_t$ together with the ground-truth next token $x_{t+1}$ to predict $x_{t+2}$.

Sequential MTP architecture

Following the V3 design, the depth-1 MTP module normalizes the main model's final hidden state $h_t$ and the shared embedding of the next token $E(x_{t+1})$, concatenates them into a $2d$-dimensional vector, and projects them back to width $d$ via a matrix $M_1$:

This representation then passes through one dedicated Transformer block $\operatorname{TRM}_1$ and is mapped to vocabulary probabilities using the main model's shared output head:

Feeding the ground-truth token $x_{t+1}$ into the MTP module during training does not leak future information into the main model's prediction of $x_{t+1}$, because $h_t$ is computed before the MTP module runs, and the MTP block's target is $x_{t+2}$. Rather than forcing $h_t$ to guess two independent future tokens simultaneously, conditioning on $x_{t+1}$ lets the auxiliary block evaluate how well $h_t$ prepares the representation for the step after $x_{t+1}$.

Training objective and speculative decoding

The total pre-training loss combines standard next-token cross-entropy with the depth-1 MTP cross-entropy:

where the weighting coefficient $\lambda_{\mathrm{MTP}}$ is set to $0.3$ during the main pre-training phase and reduced to $0.1$ once learning-rate decay begins.

At inference time, the MTP head can either be discarded—incurring zero serving overhead—or used as a built-in draft head for speculative decoding. When drafting, the main trunk emits a candidate token $\hat{x}_{t+1}$, the MTP module feeds $\hat{x}_{t+1}$ into $\operatorname{TRM}_1$ to propose $\hat{x}_{t+2}$, and the next forward pass of the main model verifies both tokens in parallel.

Sources: main.tex, Designs Inherited from DeepSeek-V3 (lines 290–305), Model and Training Setups (lines 927–985); inherited auxiliary-loss-free balancing and sequential MTP architecture from DeepSeek-V3 (papers/dsv3/main.tex, lines 358–384, 410–457).

Continue with CSA compression or return to the contents.

5. How CSA Compresses Memory

Compressed Sparse Attention (CSA) operates in two stages: first, it condenses the sequence of token hidden states into a shorter sequence of compressed key–value entries at a 4:1 stride; second, it uses a lightweight indexer to select at most $k$ of those compressed entries for each query to read. This chapter derives the first stage—the overlapping, coordinate-wise learned pooling mechanism that builds CSA's compressed memory.

The four compression projections

Given a sequence of input hidden states

CSA projects $H$ into two candidate content matrices ($C^a, C^b$) and two corresponding compression-logit matrices ($Z^a, Z^b$):

Each of the four projection matrices $W^{aKV}, W^{bKV}, W^{aZ}, W^{bZ}$ has shape $d \times c$, producing outputs in $\mathbb{R}^{N \times c}$ with head dimension $c = 512$ in both V4-Flash and V4-Pro. In addition, the compressor maintains two learned positional-bias tables $B^a, B^b \in \mathbb{R}^{m \times c}$, where $m = 4$ is the CSA compression stride.

Coordinate-wise softmax pooling

Rather than assigning a single scalar importance weight to an entire token vector, CSA computes pooling weights independently for each of the $c = 512$ coordinates.

To form compressed entry $i$ (using zero-based indexing), the compressor gathers $2m = 8$ token positions across two adjacent blocks of length $m = 4$:

• The $a$ branch covers the current block, token positions $mi, \dots, m(i+1)-1$.
• The $b$ branch covers the previous block, token positions $m(i-1), \dots, mi-1$.

For each coordinate $r \in \{0, \dots, c-1\}$, the unnormalized pooling logit $u_{j,r}$ at source token $j$ adds the intra-block positional bias to the projected logit:

A softmax across those $2m = 8$ source positions normalizes the weights for coordinate $r$:

Finally, coordinate $r$ of the compressed entry $C^{\mathrm{Comp}}_i \in \mathbb{R}^c$ is the weighted combination of the $a$-branch and $b$-branch content values:

Because the softmax runs along the sequence dimension (over the $2m$ tokens in the two-block window) independently for each channel $r$, coordinate $0$ can place nearly all of its weight on the first token in the window while coordinate $1$ focuses on the seventh token.

Worked single-coordinate example

To see how the coordinate-wise pooling operates, take a single coordinate $r$ with compression factor $m = 2$ (so each window spans $2m = 4$ tokens). Suppose the two current-block content values in $C^a$ are $10$ and $20$, and the two previous-block content values in $C^b$ are $2$ and $6$.

If adding the positional biases $B^a$ and $B^b$ yields the four logits

exponentiating gives $(2, 1, 1, 1)$, which normalizes to the softmax weights

The resulting scalar stored for coordinate $r$ is

Repeating this calculation in parallel across all $512$ coordinates produces the complete vector $C^{\mathrm{Comp}}_i$.

Receptive field versus compression stride

With $m = 4$, each interior compressed entry pools across eight source tokens, yet the sequence length still shrinks by a factor of four:

Here the receptive field of each entry is $2m = 8$ tokens, while the stride between successive entries is $m = 4$ tokens. Because a new entry is emitted only once every four tokens, a sequence of length $N$ produces $\lfloor N/4 \rfloor$ completed entries. At the very first block ($i = 0$), where no previous block exists, the $b$-branch content is zeroed and its logits are set to $-\infty$ so that the softmax distributes $100\%$ of the weight across tokens $0$–$3$.

Overlapping the two branches prevents hard block boundaries from severing multi-token expressions (such as a function name on token 3 and its opening parenthesis or argument on token 4). Moreover, because every 4-token span participates first as a current block via $(W^{aKV}, W^{aZ})$ and then as a previous block via $(W^{bKV}, W^{bZ})$, the model can extract different features from the same tokens depending on whether they sit in the leading or trailing half of the 8-token window.

Causal availability and the local sliding window

In an autoregressive decoder, token $t$ cannot read any compressed entry whose receptive field includes future positions $j > t$. If compressed block $s$ summarizes token positions $ms$ through $m(s+1)-1$, that block only becomes complete once token $m(s+1)-1$ has been processed. In the paper's notation, the indexer restricts query $t$ to completed blocks satisfying

This causal boundary must be enforced both when the indexer ranks candidate blocks and when the main attention heads read them; allowing an incomplete block into top-$k$ selection would leak future tokens into the ranking decision.

Because the currently accumulating block (and the most recent completed boundary) cannot be accessed through the causal compressed cache, queries would be blind to immediately preceding tokens if they relied on $C^{\mathrm{Comp}}$ alone. CSA bridges this gap by combining the selected compressed entries with an uncompressed local sliding window of up to 128 recent tokens, ensuring full-resolution access to local context while older tokens transition smoothly into the 4:1 compressed cache.

Decode-time cache state

During token-by-token decoding, the serving runtime must track two distinct types of state for each CSA layer:

1. The growing compressed cache: Completed 512-dimensional main KV entries $C^{\mathrm{Comp}}_i$ (plus 128-dimensional compressed indexer keys), appending one new entry every four decode steps.
2. Bounded tail buffers: The recent uncompressed tokens required for the 128-token local sliding window, along with the $b$-branch projections from the just-completed 4-token block and the partially accumulated tokens of the current block.

As Chapter 13 details, separating the append-only compressed blocks from the fixed-size rolling tail buffers allows the inference server to page long-context history efficiently without re-running the compressor on older tokens.

Sources: main.tex, Compressed Key-Value Entries under CSA (lines 389–407), causal masking and sliding window (lines 515–521), cache management (lines 831–875).

Continue with CSA selection and reading or return to the contents.

6. How CSA Selects and Reads Memory

At a one-million-token context length, 4:1 sequence compression still leaves roughly $250{,}000$ compressed entries per CSA layer. Having all 64 or 128 main query heads attend densely to $250{,}000$ vectors of width $512$ on every token would remain prohibitively expensive. CSA avoids this by splitting attention into a fast lightning indexer that scores all eligible compressed blocks in low precision, followed by shared key–value MQA over only the top-$k$ selected entries.

Low-rank query generation

For a token at position $t$ with hidden state $h_t \in \mathbb{R}^d$, CSA first projects $h_t$ down to a shared low-rank query latent $c_t^Q \in \mathbb{R}^{d_c}$:

where $d_c = 1{,}024$ in V4-Flash and $d_c = 1{,}536$ in V4-Pro. Both the lightning indexer queries and the main attention queries are expanded from this common latent vector:

Here $q_t^I$ produces $n_h^I = 64$ indexer query heads of dimension $c^I = 128$, while $q_t$ produces $n_h$ main query heads ($64$ in Flash, $128$ in Pro) of dimension $c = 512$. Sharing $c_t^Q$ amortizes the initial down-projection from width $d$ across both systems.

The lightning indexer

To score candidate compressed blocks without touching the 512-dimensional main cache, the lightning indexer maintains its own set of 128-dimensional compressed keys $K_s^{I\mathrm{Comp}} \in \mathbb{R}^{128}$, constructed using the same 4:1 overlapping pooling structure described in Chapter 5 but with separate learned projection matrices of width $c^I = 128$.

For each token $t$, a small linear projection computes a vector of per-head combiner weights $w_t^I \in \mathbb{R}^{n_h^I}$ directly from the hidden state:

The scalar index score $I_{t,s}$ between query token $t$ and preceding compressed block $s < \lfloor t/m \rfloor$ is then computed by taking the dot product of each indexer query head $q_{t,h}^I$ with the shared compressed indexer key $K_s^{I\mathrm{Comp}}$, passing the matches through a ReLU activation, and summing them with weights $w_{t,h}^I$:

Replacing a softmax with $\operatorname{ReLU}$ avoids expensive exponentiation and normalization over $250{,}000$ candidates during the scan, and the entire indexer evaluation is trained via quantization-aware training to execute in FP4.

Worked ranking example

To see how the head combiner weights $w_t^I$ steer block selection, consider a two-head toy indexer scoring three candidate blocks:

If token $t$ produces combiner weights $w_t^I = (0.8, 0.2)$, the resulting block scores are

so a top-2 selector picks blocks A and C. If the next token shifts its combiner weights to $w_t^I = (0.2, 0.8)$, the scores become $I_A = 1.8$, $I_B = 5.0$, and $I_C = 2.2$, elevating block B to first place even though the underlying keys $K_s^{I\mathrm{Comp}}$ are shared across all heads.

Top-$k$ selection and asymptotic scaling

Once the indexer scores all eligible blocks $s < \lfloor t/m \rfloor$, it selects the indices of the top $k$ blocks ($k = 512$ in V4-Flash, $k = 1{,}024$ in V4-Pro). Note that $k$ counts compressed entries, not raw tokens: selecting $1{,}024$ entries in V4-Pro retrieves $1{,}024$ summary vectors produced at stride 4 (each with an 8-token receptive field), and those selected entries are shared across all $n_h$ main query heads for token $t$.

Splitting selection from reading divides per-token CSA decode work into two distinct terms:

1. The indexer scan over all $\approx N/m$ compressed blocks, which still scales linearly with context length $N$: $$O\left(n_h^Ic^I\frac{N}{m}\right).$$
2. Main attention over the $k$ selected compressed blocks plus the $n_{\mathrm{win}} = 128$ uncompressed sliding-window entries, which is capped at a constant budget once $N/m \ge k$: $$O\left(n_hc(k+n_{\mathrm{win}})\right).$$

Thus, CSA does not make decode strictly $O(1)$ or prefill strictly $O(N)$ in sequence length; rather, it pushes the $N$-dependent scan into a narrow ($c^I=128$), low-precision (FP4) indexer while bounding the expensive 512-dimensional main attention pass at $k + 128$ entries per token.

Shared key–value MQA

After top-$k$ selection gathers the chosen main compressed vectors $C_t^{\mathrm{selected}}$, each main query head $q_{t,h}$ attends over those entries (together with the local sliding window) using shared key–value MQA:

Here every query head $h \in \{1, \dots, n_h\}$ reads from the same shared pool of vectors $C_t^{\mathrm{selected}}$, and each vector serves simultaneously as the key $K$ and the value $V$. Compared to standard MQA—which shares a key head and a value head across query heads but still stores separate $K$ and $V$ tensors—tying $K$ and $V$ cuts the main compressed cache payload in half again.

Grouped output projections

Because each of the $n_h$ query heads produces a 512-dimensional output vector $o_{t,h}$, concatenating all head outputs yields a vector of width $64 \times 512 = 32{,}768$ in V4-Flash and $128 \times 512 = 65{,}536$ in V4-Pro. Projecting that concatenated vector directly back to the model's hidden dimension $d$ with a single dense matrix $W^O \in \mathbb{R}^{(n_h c) \times d}$ would require $c n_h d$ parameters per layer.

Instead, DeepSeek-V4 partitions the $n_h$ heads into $g$ groups of 8 heads each ($g = 8$ in Flash, $g = 16$ in Pro). Within each group, the $8 \times 512 = 4{,}096$ concatenated head coordinates are projected down to an intermediate bottleneck of width $d_g = 1{,}024$, and the $g$ intermediate vectors are then concatenated and projected to width $d$.

This two-stage factorization reduces the output projection parameter count to

For V4-Flash ($n_h = 64, c = 512, d = 4{,}096, g = 8, d_g = 1{,}024$), comparing the direct and grouped parameter counts gives:

cutting the projection weights and FLOPs by exactly 50%.

For V4-Pro ($n_h = 128, c = 512, d = 7{,}168, g = 16, d_g = 1{,}024$):

reducing the output projection to 39.3% of the direct matrix size (a 60.7% reduction, saving $\approx 285\text{M}$ parameters per attention layer). Mathematically, because no nonlinearity sits between the two matrices, each group projection is a low-rank factorization that compresses 4,096 head channels through a rank-1,024 subspace before mixing across groups.

Training the discrete selector

Because $\operatorname{TopK}$ is a discrete selection operator, gradients from the main attention output flow into the selected memory entries $C_t^{\mathrm{selected}}$ and their compressor projections, but do not naturally backpropagate through the discrete index set to train the lightning indexer's projections ($W^{IUQ}, W^w$, and the indexer key compressor).

Following the DeepSeek Sparse Attention (DSA) training recipe introduced in V3.2, V4 pre-trains CSA in stages: the model first warms up with dense attention over the compressed entries, then runs a brief lightning-indexer warmup phase where the indexer is trained to match the main attention's importance distribution, and finally switches to sparse top-$k$ attention for the remainder of pre-training and context extension. Chapter 11 covers how this schedule fits into the 32T/33T-token pre-training run.

Sources: main.tex, Lightning Indexer, Shared Key-Value MQA, and Grouped Output Projection (lines 409–449); sparse attention training schedule (lines 961–985).

Continue with HCA and positional details or return to the contents.

7. HCA, Local Attention, and Position

While Compressed Sparse Attention (CSA) preserves fine-grained 4:1 resolution and relies on a sparse top-$k$ selector, Heavily Compressed Attention (HCA) takes the opposite tack: it pools 128 consecutive tokens into a single summary entry ($m' = 128$) and attends densely to every eligible entry in the compressed sequence. Alongside HCA, both attention types share four critical architectural mechanisms—a 128-token uncompressed local window, query/key RMSNorm, partial Rotary Position Embeddings with output de-rotation, and learned attention sinks.

HCA's 128:1 non-overlapping compressor

Given input hidden states $H \in \mathbb{R}^{N \times d}$, an HCA layer projects $H$ into a single content matrix $C$ and a single compression-logit matrix $Z$:

where $W^{KV}, W^Z \in \mathbb{R}^{d \times c}$ with $c = 512$, and adds a learned intra-block positional bias $B \in \mathbb{R}^{m' \times c}$ with $m' = 128$.

Unlike CSA, HCA does not overlap adjacent blocks. Each block $i$ covers the disjoint span of $m' = 128$ tokens from $j = m'i$ to $m'(i+1)-1$. Applying a coordinate-wise softmax over those 128 positions produces weight vectors $S_j \in \mathbb{R}^c$, which pool the block's content into a single compressed entry:

At a context length of $N = 1{,}000{,}000$ tokens, HCA produces $\lfloor 1{,}000{,}000 / 128 \rfloor = 7{,}812$ completed compressed entries (with 64 trailing tokens covered by the local sliding window).

Dense attention over heavily compressed history

Because the entire million-token history shrinks to just $7{,}812$ entries, HCA eliminates the lightning indexer altogether. Instead, HCA expands $n_h$ query heads ($64$ in Flash, $128$ in Pro) from a low-rank query latent $c_t^Q$, reads all eligible compressed blocks $i < \lfloor t / 128 \rfloor$ plus the local sliding window via shared key–value MQA, and projects the head outputs back to width $d$ using the same grouped output projection structure as CSA.

Comparing the two attention layers highlights how they trade sequence resolution against selection coverage:

Notice that at a one-million-token context in V4-Pro, a single HCA layer actually performs more main-attention reads per token ($7{,}812$ entries) than a CSA layer ($1{,}024$ entries). HCA's efficiency advantage lies in its tiny KV cache footprint (one 512-dimensional vector per 128 tokens) and the complete absence of an indexer scan. Interleaving CSA and HCA across the network allows HCA layers to maintain broad, uninterrupted awareness of the entire context trajectory while CSA layers retrieve specific 4-token passages on demand.

The 128-token uncompressed sliding window

Both CSA and HCA augment their compressed historical entries with up to $128$ recent uncompressed KV vectors inside core attention. For HCA, whose compressed blocks only become causally available once a full 128-token chunk finishes, a 128-token local window is the minimum span required to ensure that every token between the last completed block boundary and the current position $t$ remains directly visible.

Rather than running a separate local attention module and summing two outputs, V4 concatenates the recent uncompressed KV entries with the compressed entries (all $i < \lfloor t/m' \rfloor$ in HCA, or the top-$k$ selected entries in CSA) so that each query head computes a single unified softmax over both compressed history and immediate local context. Note that because the local window spans 128 tokens while CSA completes a new compressed block every 4 tokens, recently completed blocks can appear simultaneously in the local uncompressed window and in the candidate pool of compressed entries.

Query and KV RMSNorm

Prior to computing attention dot products, V4 applies RMSNorm independently to every query head $q_{t,h}$ and to the shared KV vector. Bounding the $L_2$ norm of queries and keys prevents dot-product logits from growing unboundedly as weight matrices scale up during training. As the paper notes in its discussion of the Muon optimizer (Chapter 10), normalizing queries and keys directly inside the attention layer keeps attention logits stable without requiring external weight-clipping heuristics such as QK-Clip.

Partial RoPE and output de-rotation

To encode positional relationships, DeepSeek-V4 uses partial Rotary Position Embeddings (RoPE): out of the $c = 512$ coordinates in each query head and shared KV vector, only the last 64 coordinates receive rotary position embeddings, while the first 448 coordinates remain position-invariant content channels.

Let $R_t$ denote the block-diagonal orthogonal rotation matrix for position $t$ acting on the 64 rotary coordinates. When scoring query $t$ against entry $s$, the rotary subspace contributes

which depends cleanly on the relative offset $s - t$.

However, because V4 ties keys and values into a single shared vector, the stored vector $v_s$ has its last 64 coordinates rotated by $R_s$. When attention forms the weighted sum of values across source positions $s$, the rotary portion of the head output is

Left uncorrected, $o_t$ would carry absolute positional rotations $R_s$ into the output projection, making the representation vary with absolute sequence index rather than relative distance. DeepSeek-V4 resolves this by applying the inverse query rotation $R_{-t} = R_t^\top$ to the rotary coordinates of the attention output:

After this output de-rotation, every attended value's rotary coordinates are expressed strictly in terms of its relative position $s - t$ with respect to the query token $t$.

Learned attention sinks

In standard softmax attention, the attention weights across all accessible entries must sum to $1$, forcing a head to allocate $100\%$ of its probability mass even when no historical entry matches the query. DeepSeek-V4 adds a single learned sink logit $z'_h \in \mathbb{R}$ per query head to the softmax denominator:

Mathematically, adding $\exp(z'_h)$ to the denominator is equivalent to appending a virtual key with logit $z'_h$ whose corresponding value vector is zero. If the real candidate entries have total exponential mass $\sum_j \exp(z_{h,t,j}) = 4$ and the sink has $\exp(z'_h) = 6$, the weights assigned to the real entries sum to $4 / (4 + 6) = 0.4$, scaling down the head's output rather than injecting noise from irrelevant context blocks. Unlike prompt-based "attention sink" workarounds that rely on attending to the initial BOS token, a learned denominator logit requires no extra KV cache entry or memory read.

Sources: main.tex, Heavily Compressed Attention (lines 461–496), Other Details including sliding window, QK RMSNorm, partial RoPE with inverse rotation, and attention sinks (lines 497–531).

Continue with the efficiency model or return to the contents.

8. A Cost Model for Long Context

DeepSeek-V4 attacks long-context overhead across several distinct dimensions: KV cache footprint, main-attention memory reads, lightning-indexer scan arithmetic, output projection matrix sizes, and FP4 expert execution. Because each of these savings scales differently with sequence length $N$ and batch size, this chapter builds a first-principles cost model from the reported dimensions of V4-Flash and V4-Pro, connects those numbers to the paper's comparisons against GQA8 and DeepSeek-V3.2, and examines where memory bandwidth and kernel launch overheads separate FLOP reductions from wall-clock latency.

Separating inference memory categories

An inference server's GPU memory footprint divides into four distinct pools:

1. Model weights: Fixed once the checkpoint is loaded across the cluster (though substantial for a 284B or 1.6T MoE model).
2. Growing KV caches: Scales linearly with sequence length $N$ and the number of concurrent requests.
3. Bounded per-request tail buffers: Fixed-size buffers holding the 128-token uncompressed sliding window and unfinished compression block states.
4. Runtime workspace: Temporary activation buffers and communication staging areas that depend on batch size and kernel implementation.

Shrinking the growing KV cache allows a cluster that already holds the model weights to serve many more concurrent million-token sessions without exhausting HBM.

Baseline: conventional GQA8 cache payload

To establish a reference point, consider a conventional Grouped-Query Attention baseline with 8 KV heads (GQA8), head dimension $128$, and BF16 ($2$-byte) storage. Storing separate key and value vectors requires

per token per layer. Across a sequence of $N = 1{,}000{,}000$ tokens, a single layer consumes $4.096\text{ GB}$ of KV cache:

• Over 61 layers (matching V4-Pro's depth), a GQA8 cache holds $249.856\text{ GB}$ per million-token request.
• Over 43 layers (matching V4-Flash's depth), a GQA8 cache holds $176.128\text{ GB}$ per million-token request.

V4 compressed-entry payload

DeepSeek-V4 stores each 512-dimensional shared KV entry in a mixed-precision format: the 448 non-rotary content channels are stored in FP8 ($1\text{ byte}$ per value), while the last 64 rotary (RoPE) channels are kept in BF16 ($2\text{ bytes}$ per value) to preserve positional phase precision. Because the same vector acts as both key and value, a single 512-dimensional entry occupies

Dividing by the sequence compression stride gives the amortized main-cache growth rate per original token:

• CSA main entry ($m = 4$): $576 / 4 = 144\text{ bytes}$ per original token.
• HCA main entry ($m' = 128$): $576 / 128 = 4.5\text{ bytes}$ per original token.

In addition to its main KV entry, each CSA layer caches a 128-dimensional compressed key for the lightning indexer. Stored in FP4 ($0.5\text{ bytes}$ per scalar before quantization scale metadata), each indexer key occupies $128 \times 0.5 = 64\text{ bytes}$ per 4-token block, adding $64 / 4 = 16\text{ bytes}$ per original token. Combining the main entry and indexer key gives a raw scalar payload of $160\text{ bytes}$ per token in a CSA layer.

Whole-model KV cache footprint

Using the layer schedules from Chapter 3—where V4-Pro interleaves 30 CSA and 31 HCA layers, and V4-Flash uses 2 pure local-window layers followed by 21 CSA and 20 HCA layers—the raw growing cache payload for $N$ tokens is:

and

At $N = 1{,}000{,}000$ tokens, this comes to $4.94\text{ GB}$ for V4-Pro and $3.45\text{ GB}$ for V4-Flash. Divided by the layer-matched GQA8 baselines ($249.86\text{ GB}$ and $176.13\text{ GB}$), both models sit right at $1.98\%$—matching the paper's statement that V4's KV cache requires roughly $2\%$ of a standard GQA8 cache.

Quantization scales and bounded tail buffers

In an actual serving engine, per-request memory exceeds these raw scalar payloads for two reasons:

1. Quantization scale factors: Block-quantized FP8 and FP4 tensors store scaling metadata alongside the packed values. For instance, if a 128-dimensional FP4 indexer key ($64\text{ bytes}$) stores one 1-byte scale factor per 32 elements, the scales add $4\text{ bytes}$ ($+6.25\%$) per block.
2. Sliding-window and unfinished-block tails: Each layer must maintain the 128-token uncompressed sliding window plus the rolling state for incomplete compression blocks (the $b$-branch buffer in CSA or up to 127 uncommitted tokens in HCA). At $N = 1{,}000{,}000$, these fixed buffers are negligible compared to the $4.94\text{ GB}$ growing cache, but at short contexts ($N \le 2{,}048$), bounded window and tail buffers make up a large fraction of the per-request footprint.

Main-attention versus indexer arithmetic

During single-token decoding, an attention layer with $n_h$ query heads of dimension $c$ attending to $E$ entries performs $2cE$ FLOPs per head for query–key dot products and $2cE$ FLOPs per head for weighted value accumulation, totaling

pairwise operations per layer (excluding projections, RMSNorm, and softmax).

In V4-Pro ($n_h = 128, c = 512$) at $N = 1{,}000{,}000$ tokens:

• CSA main attention reads at most $k = 1{,}024$ selected compressed entries plus $128$ local window tokens, so $E = 1{,}152$: $$4 \times 128 \times 512 \times 1{,}152 \approx 3.02 \times 10^8\text{ FLOPs per CSA layer}.$$
• HCA main attention reads all $7{,}812$ completed blocks plus $128$ local window tokens, so $E = 7{,}940$: $$4 \times 128 \times 512 \times 7{,}940 \approx 2.08 \times 10^9\text{ FLOPs per HCA layer}.$$

Now compare CSA's main attention against its lightning indexer, which must score all $N/4 = 250{,}000$ candidate blocks using $n_h^I = 64$ heads of dimension $c^I = 128$:

At one million tokens, the CSA indexer performs over $13\times$ more raw scalar operations ($4.10\text{B}$) than CSA's main attention heads ($302\text{M}$). This is why running the lightning indexer in FP4—where tensor cores execute four-bit dot products at higher throughput and read $4\times$ fewer bytes than BF16—is essential to keeping CSA fast at million-token lengths.

Prefill scaling behavior

During a full-sequence prefill of length $N$ (without prefix-cache hits), query $t$ attends over its causal prefix:

• In HCA, query $t$ reads $\approx t/128$ compressed entries, so summing across $t = 1, \dots, N$ yields pairwise attention work proportional to $\frac{1}{128} \cdot \frac{N^2}{2}$.
• In CSA, the lightning indexer scans $\approx t/4$ compressed keys at step $t$, yielding pairwise indexer work proportional to $\frac{1}{4} \cdot \frac{N^2}{2}$ (executed in narrow 128-dim FP4), while the main 512-dim attention heads remain linear ($O(Nk)$) once $t/4 > k$.

Thus, while sequence compression and sparse selection shrink the quadratic constant by more than an order of magnitude, full prefill over uncached prompts still retains an $O(N^2)$ component from the HCA dense pass and the CSA indexer scan.

Reconciling the V3.2 comparison

As noted in Chapter 1, the paper reports that at a 1M-token context, V4-Pro requires 27% of DeepSeek-V3.2's single-token inference FLOPs and 10% of its KV cache, while V4-Flash requires 10% of V3.2's FLOPs and 7% of its KV cache.

Why is V4's KV cache $\approx 2\%$ of a GQA8 baseline, yet $7\%$–$10\%$ of DeepSeek-V3.2? Because V3.2 already used Multi-Head Latent Attention (MLA) with DeepSeek Sparse Attention (DSA), which compressed KV heads into a shared low-rank latent vector per token. V3.2 did not, however, compress along the sequence dimension; every token still stored its own MLA latent vector and DSA indexer key. V4's additional $10\times$ cache reduction over V3.2 comes directly from 4:1 CSA and 128:1 HCA sequence pooling plus FP8/FP4 entry storage.

In the FLOP comparison, the paper converts all arithmetic into equivalent FP8 FLOPs (discounting FP4 operations by half). The authors also note a hardware caveat: on current GPU generations, mixed-precision FP4 × FP8 matrix instructions share the same peak throughput as FP8 × FP8, whereas native hardware support for asymmetric FP4 × FP8 throughput could reduce the effective cost by an additional third.

Why FLOP ratios do not translate 1:1 into wall-clock latency

In production serving, wall-clock token latency often diverges from raw FLOP counts across several bottlenecks:

Because each regime stresses a different hardware limit, Chapters 12 and 13 examine how V4's kernel and caching stacks turn theoretical byte and FLOP savings into actual throughput.

Sources: main.tex, efficiency comparisons (lines 217–227, 533–545), model dimensions (lines 927–957), FP4 throughput analysis (lines 1246–1264).

Continue with mHC or return to the contents.

9. Manifold-Constrained Hyper-Connections

In a standard Transformer, a single residual vector of width $d$ carries token state from layer to layer. Manifold-Constrained Hyper-Connections (mHC) widen the residual highway into $n_{\mathrm{hc}} = 4$ parallel streams of width $d$ per token while collapsing those streams back to a single vector of width $d$ before each attention or MoE sublayer. To prevent unconstrained cross-stream mixing matrices from exploding or vanishing across 61 layers, mHC projects each residual mixing matrix onto the Birkhoff polytope of doubly stochastic matrices via Sinkhorn–Knopp normalization.

From a single residual stream to $n$ streams

In a conventional pre-norm residual block, layer $l$ updates the token state $x_l \in \mathbb{R}^{1 \times d}$ by adding the sublayer transformation $F_l(x_l)$:

With $n = n_{\mathrm{hc}} = 4$ streams, each token carries a matrix of residual states

and the residual update around sublayer $F_l$ (either an attention module or an MoE module) takes the generalized form

The three learned coefficient objects play distinct structural roles:

Because $A_l X_l \in \mathbb{R}^{1 \times d}$ has the standard hidden width $d$, the expensive attention projections and MoE experts run at exactly the same width as in a single-stream Transformer. When $n = 1$ and $A_l = B_l = C_l = 1$, the update reduces to the standard residual connection.

Why unconstrained hyper-connections destabilize deep networks

Unrolling the direct residual path across $L$ sublayers yields a product of mixing matrices:

In unconstrained Hyper-Connections, the entries of $B_l$ are unrestricted real numbers. If the largest singular value $\sigma_{\max}(B_l)$ exceeds $1$ by even a few percent per sublayer, multiplying across $2 \times 61 = 122$ sublayers in V4-Pro amplifies activations and backpropagated gradients exponentially; conversely, eigenvalues below $1$ or negative entries cause signal attenuation and destructive cancellation. The DeepSeek-V4 authors report that naive unconstrained hyper-connections trigger severe numerical instability at scale, motivating a hard geometric constraint on $B_l$.

Doubly stochastic matrices and the Birkhoff polytope

mHC constrains every residual mixing matrix $B_l$ to be doubly stochastic—meaning all entries are nonnegative and every row and column sums to $1$:

Geometrically, the set of $n \times n$ doubly stochastic matrices forms the Birkhoff polytope (a convex polytope whose vertices are the $n!$ permutation matrices; despite the name "manifold-constrained," the feasible region is a polytope with boundary faces rather than a boundaryless smooth manifold).

Row stochasticity ($B\mathbf{1} = \mathbf{1}, B \ge 0$) ensures that each updated stream is a convex combination of the incoming streams, so for any coordinate $r \in \{1, \dots, d\}$, the mixed values in $B_l X_l$ always lie between the minimum and maximum values of the incoming streams. Column stochasticity ($\mathbf{1}^\top B = \mathbf{1}^\top$) ensures that the sum of activations across the $n$ streams is exactly conserved at every coordinate.