Introduction

DeepSeek-V4.1-Flash is designed around the economics of sustained, input-heavy agent workloads. Its technical report brings together a causal encoder-decoder (CED), cross-layer attention memory sharing, hierarchical sparse retrieval, 4-bit caches, bounded local-state replay, conditional parameter memory (Engram), native vision pre-training, and large-scale agent post-training. Each mechanism targets a distinct bottleneck in prefill compute, live HBM footprint, persistent prefix storage, or autoregressive decoding; this book examines how those mechanisms work individually and how they interact across the full system.

The primary source is DeepSeek-AI's DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression, a 51-page technical report from the model repository (archived alongside this book at papers/dsv41/DeepSeek_V41_Tech_Report.pdf), supplemented by the released configuration, reference inference code, prompt-encoding templates, and evaluation scripts. Chapter 29 catalogues the source manifest, checksums, revisions, and discrepancies between the prose report and the public code.

While the report provides unusually rich systems detail, several pieces required for a complete replication remain undisclosed—including exact task inventories, fine-grained data mixture ratios, production kernel implementations, hardware cost breakdowns, and isolated ablations for every architectural change. Where the report leaves gaps, the chapters below derive the implied mechanics from the released dimensions and equations and note clearly what remains unspecified.

Contents

Suggested reading paths

For a fast overview of the main contributions, read Chapters 1–3, 6, 8–12, 22, 24, and 28. Readers focused on model architecture can follow Chapters 2–16 and 27; those focused on training and data can follow Chapters 4–5 and 16–22. Systems and serving engineers should focus on Chapters 6–15, 18, 21, and 26, while evaluation and benchmark analysis are covered in Chapters 22–25 and 28.

Conventions

Layer identifiers are zero-indexed unless explicitly referred to as ordinal positions: layer 20 is the twenty-first backbone layer and the first decoder layer. The backbone has 40 layers, and the DSpark speculative drafting module adds three draft blocks outside that count.

Storage calculations use decimal gigabytes (GB) and terabytes (TB). "One million tokens" in worked calculations denotes $1{,}000{,}000$ sequence positions (the released max_position_embeddings limit is $1{,}048{,}576$), and visual embeddings occupy sequence positions alongside text and protocol tokens. Unless noted otherwise, global KV cache figures exclude sliding-window local state, model weights, allocator fragmentation, scratch buffers, and serving metadata.

1. The Workload and the Problem

A long-running software or research agent alternates between emitting short actions and ingesting large blocks of new environment feedback. It reads source files, executes shell commands, inspects test suites, opens documentation, and parses tool outputs, building up a transcript that functions as a stateful working environment. DeepSeek-V4.1-Flash is architected specifically to drive down the compute, memory, and storage costs of maintaining and extending that growing context.

Cutting attention FLOPs alone does not make long-horizon agent workloads cheap. Prefill compute across non-attention layers, live key-value (KV) footprint in accelerator memory, and the storage and bandwidth required to persist prefixes across turns all become bottlenecks in long multi-turn sessions. V4.1-Flash targets computation, stored state, and data movement together.

Why agent input can outweigh output

Consider an agent that emits a 200-token tool call to run a build and receives a 20,000-token compiler log in return. Deciding on the next action might take only a few hundred generated tokens, yet the serving system must first ingest and encode the entire 20,000-token log. Over dozens of turns, the transcript accumulates repository trees, failing stack traces, UI screenshots, and prior patch attempts. Unlike a single-turn math proof where output generation dominates, interactive repository and browser agents typically process far more input tokens than they generate.

The actual cost of each turn hinges on prefix caching. When the serving infrastructure matches an incoming transcript prefix against cached state, it only needs to run prefill on the newly appended suffix. If that cached prefix has been evicted from DRAM or SSD, or if the model architecture makes cached state too bulky to retain across pauses between tool calls, the cluster must recompute part or all of the historical transcript from scratch. Sustaining long agent sessions therefore requires both fast incremental ingestion and compact, reusable prefix state.

Three distinct cost centers

The report's headline efficiency numbers target three separate system resources that are easy to conflate:

1. Prefill computation is the arithmetic required to transform incoming tokens into hidden states and cache entries. Standard causal Transformers run every layer—both attention and feed-forward Mixture-of-Experts (MoE) blocks—over the entire input sequence.
2. Runtime context memory is the working state an active request holds in accelerator high-bandwidth memory (HBM) during generation. Even when GPU compute units are underutilized, HBM capacity caps how many long-context requests a node can serve concurrently.
3. Persistent prefix memory is the serialized cache state retained in host DRAM or solid-state drives (SSDs) across turns while the agent waits on external tools or user input. Reusing this state depends on both storage capacity (which governs eviction rates) and interconnect bandwidth (which governs how fast cached prefixes can be staged back into HBM).

When the report notes that V4.1-Flash roughly halves prefill compute, shrinks the active global KV cache to about one-quarter of V4-Flash, and reduces persistent KV storage to roughly one-eighth, each ratio applies to a different stage of the serving lifecycle.

Why sparse attention is only part of the solution

In dense causal attention, every new query compares itself against all preceding keys and aggregates their values. Sparse attention restricts the heavy value aggregation to a small subset of selected positions, dramatically reducing main-attention arithmetic at long context lengths.

Even when sparse attention cuts the cost of reading values, however, several large costs remain untouched:

• Index scanning: A lightweight indexer still has to score past keys to decide which entries the main attention head will read, and that scan grows with context length unless the candidate pool is also restricted.
• Memory footprint: Unselected KV entries cannot simply be discarded; they must remain in memory in case a future token needs to attend to them. Sparse reads alone do not shrink the KV cache by a single byte.
• Non-attention work: Routed and shared MoE experts, multi-stream residual traffic, vision encoding, and all-to-all expert-parallel communication execute across every active layer regardless of how sparse the attention kernel is.

To reduce end-to-end cost, V4.1-Flash pairs sparse attention with 4-bit cache quantization, cross-layer KV bank sharing, cross-layer index reuse, a causal encoder-decoder split that skips decoder layers during bulk prefill, and bounded local-state replay so sliding-window caches do not have to be persisted to disk.

Modeling interaction cost

For a single agent turn that ingests $I$ new input positions and generates $O$ output tokens, the end-to-end latency or resource cost decomposes into five terms:

Prefix cache hits reduce the effective input length $I$ that must undergo prefill, while cache misses or state reconstruction introduce $C_{\mathrm{replay}}$ and host-to-device transfer cost $C_{\mathrm{transfer}}$, and external sandbox execution accounts for $C_{\mathrm{tools}}$.

Because each architectural optimization acts on only one term in this sum, end-to-end speedups follow Amdahl's law rather than multiplying directly. Halving the per-token prefill cost $c_{\mathrm{prefill}}$ matters most when $I \gg O$; shrinking the persistent cache footprint increases the cache hit rate (keeping effective $I$ small across turns) and cuts $C_{\mathrm{transfer}}$; speculative decoding via DSpark and reasoning-effort control target $O c_{\mathrm{decode}}$. If prefill accounts for 60% of model inference time on a given workload and all other components remain constant, cutting prefill work in half reduces normalized inference time to $0.6/2 + 0.4 = 0.7$, a $1.43\times$ end-to-end speedup.

Memory footprint as a concurrency bottleneck

At the reported global cache footprint of 890 bytes per token position, a single 1,000,000-token context consumes $0.89\text{ GB}$ of global KV memory. Serving 100 concurrent sessions at that length requires $89\text{ GB}$ of HBM for global KV entries alone—before accounting for model weights, 128-token local sliding-window caches, activation buffers, quantization scales, or runtime page tables.

Shaving hundreds of bytes off each token position directly increases the batch concurrency an HBM budget can hold. Meanwhile, in tiered DRAM/SSD storage, an $8\times$ smaller persistent prefix footprint means a fixed storage pool can hold eight times as many paused agent trajectories before evicting old sessions. Higher cache retention avoids expensive full-history prefill recomputation on subsequent turns, turning a storage compression win into a compute and latency reduction.

Trading representational freedom for runtime efficiency

Giving every Transformer layer its own high-precision, uncompressed KV cache and independent retrieval index maximizes representational flexibility. Sharing KV banks across layers, quantizing entries to 4-bit floating point, reusing sparse selection indices across adjacent blocks, and reconstructing local window state from a 128-token replay suffix all constrain that freedom and introduce controlled approximations.

The central engineering question of the V4.1-Flash report is whether a large MoE backbone (552B backbone parameters plus 196B Engram parameters—larger in total weight capacity than V4-Flash) can absorb these state constraints while improving task accuracy. As the report notes in its limitations discussion, aggressive sparse selection and approximate boundary replay remain sensitive to needle-in-a-haystack retrieval failures and boundary artifacts if not carefully trained with quantization-aware and replay-aware stages.

Sources: Report §1 (pp. 4–7), §3.2 (pp. 18–20), and §6 (p. 37). Chapter 2 reviews the foundational mechanics of causal attention, KV caching, MoE routing, and post-training used throughout the book.

2. Foundations: Tokens, Attention, Caches, and Learning

DeepSeek-V4.1-Flash is an autoregressive multimodal model: it maps a sequence of text tokens and visual patch embeddings into a probability distribution over the next text token. Before diving into the specific layers and compression schemes of V4.1-Flash, this chapter establishes the core mathematical and systems terminology used throughout the report.

Tokens, embeddings, and hidden states

A tokenizer segments raw text into discrete integer IDs corresponding to words, subwords, punctuation, or byte sequences, and an input embedding table maps each ID to a dense vector. As a token representation passes through the network's layers, its evolving vector is referred to as the hidden state. In V4.1-Flash, the primary hidden dimension is $d = 5{,}120$. While the model's multi-stream hyper-connection (mHC) residual pathway maintains four parallel streams per token position, heavy attention and expert projections operate on mixed single-stream representations of width $5{,}120$.

Visual inputs bypass the text tokenizer. A vision encoder (DeepSeek-ViT) and a two-layer MLP projector convert image patches into width-$5{,}120$ vectors that are spliced directly into the token sequence. Because visual embeddings and formatting protocol markers occupy sequence slots just like text tokens, they draw from the same $1{,}048{,}576$-position context budget.

Causality in encoders and decoders

A causal sequence model predicts position $t+1$ using only information from positions $\le t$, enforced via an upper-triangular attention mask that blocks queries from reading future keys. During training, the full sequence is already known, so all positions can be evaluated in parallel across matrix-multiply units while the mask preserves causality at every step.

In standard NLP terminology, "encoder" often refers to a bidirectional network (such as BERT) where every token attends to both past and future tokens. V4.1-Flash uses the term causal encoder differently: its 20-layer encoder is strictly causal and never looks ahead into future positions. What distinguishes the encoder from the 20-layer decoder is how their hidden states and KV caches are routed during prefill and generation, not their causal masking.

Scaled dot-product attention

Given a hidden state $h_t$ at position $t$, learned linear projections produce a query vector $q_t$ along with key vectors $k_j$ and value vectors $v_j$ across positions $j$. A standard causal attention head computes normalized alignment weights $a_{tj}$ over the causally visible set $\mathcal{V}_t = \{j \le t\}$ and takes a weighted sum of values:

Multi-head attention runs this comparison across multiple parallel subspaces (64 query heads of width 512 in V4.1-Flash). Importantly, every query head does not need its own separate key and value tensors in memory. Following the Multi-head Latent Attention (MLA) and Compressed Sparse Attention (CSA) lineage, V4.1-Flash stores a shared 512-channel compressed latent per cached entry, combines sparse global retrieval with a 128-token local sliding window, and incorporates an attention sink and decoupled rotary position embeddings (RoPE).

What the KV cache stores

During autoregressive generation, predicting token $t+1$ requires comparing the new query $q_t$ against keys and values from all preceding positions $1, \dots, t$. Recomputing those historical keys and values from raw token IDs at every step would make generating an $N$-token sequence cost $O(N^2)$ full forward passes. A KV cache avoids this by retaining the projected attention representations of past tokens in memory.

Unlike model weights (which are static across requests) or intermediate MLP activations (which can be discarded immediately after a layer finishes computing the current position), KV cache tensors are request-specific state whose lifetime spans the entire generation session. Any change to the model weights, quantization format, positional encoding scheme, or prefix token sequence invalidates the cached entries.

Prefill versus decode

Inference splits into two execution regimes with very different hardware bottlenecks:

• Prefill ingests the initial prompt or a newly appended multi-turn suffix (such as a tool output). Because thousands of input tokens arrive simultaneously, projections and expert weights are reused across a large batch of token positions, making prefill predominantly compute-bound.
• Decode generates new tokens one step at a time (or a few tokens at a time under speculative decoding). Each step processes a tiny token count while reading large weight matrices and scanning the accumulated KV cache, making decoding predominantly memory-bandwidth-bound and sensitive to kernel launch overhead.

Even when V4.1-Flash's causal encoder-decoder skips bulk decoder execution during prefill, the final prompt position still has to pass through the decoder stack so the model can emit the probability distribution for the first generated token.

Four axes of KV cache compression

To shrink attention memory and bandwidth, V4.1-Flash combines four distinct compression techniques that operate along orthogonal dimensions:

1. Sequence compression (token pooling): Reduces the number of stored entries along the time axis. With compression ratio $r = 2$ in the encoder, each pair of consecutive token positions is merged into a single cached latent vector.
2. Sparse selection: Reduces how many cached entries a query reads during attention. Even if a global bank holds $1{,}000{,}000$ entries, a query in V4.1-Flash retrieves at most $512$ global entries plus its 128-token local window.
3. Cross-layer sharing: Reduces the number of independent KV banks across the network's depth. Instead of storing 40 separate global caches for 40 layers, multiple layers issue their own queries against a shared KV bank (and can also share the selected index lists).
4. Low-precision quantization: Reduces the bit-width of each stored scalar (from BF16/FP8 down to FP4 with per-group scaling factors).

Each axis introduces a different trade-off: sequence compression blurs fine-grained positional boundaries, sparse selection risks missing relevant distant tokens, layer sharing forces different depths to read from a common representation space, and quantization adds numerical reconstruction noise.

Mixture-of-Experts (MoE) and parameter counts

While attention routes information across sequence positions, feed-forward networks (FFNs) transform each position's representation independently. In a Mixture-of-Experts layer, a single dense FFN is replaced by one shared expert that runs on every token alongside a bank of routed experts (384 per layer in V4.1-Flash), from which a lightweight router selects the top-$k$ ($k=6$) experts per token.Each expert is a gated SwiGLU MLP with intermediate width $2{,}304$.

Because only 1 shared and 6 routed experts execute for any single token at a given layer, total parameters (which govern disk storage and cluster-wide HBM capacity required to hold the weights) diverge sharply from active parameters (which govern per-token FLOPs and weight-read bandwidth during a single forward step).

Pre-training, SFT, RL, and on-policy distillation

Training proceeds through four major phases:

• Pre-training optimizes next-token cross-entropy loss over 45 trillion tokens of text and interleaved image-text data: $$\mathcal L=-\sum_t\log p_\theta(x_{t+1}\mid x_{\le t}).$$ When images are present in the prefix $x_{\le t}$, their projected visual embeddings condition the prediction of subsequent text tokens, though the loss itself is computed only over predicted text tokens.
• Supervised fine-tuning (SFT) trains the base model on curated demonstrations of tool use, reasoning traces, and multi-turn instructions.
• Reinforcement learning (RL) samples trajectories from the model inside executable sandboxes and updates policy weights $\theta$ using outcome rewards from automated verifiers and reward models.
• On-policy distillation (OPD) generates rollouts using the student policy and trains the student to match token-level probability distributions provided by stronger domain-specific teacher models along those student-visited states. In an asynchronous training pipeline, "on-policy" means trajectories are sampled from the student rather than a static offline dataset, even though rollouts may lag the latest parameter update by several optimization steps.

Sources: Report §§2.1–2.3 (pp. 7–12), §4.2.1 (pp. 21–22), and released config.json. Chapter 3 maps out how these components fit together in the 40-layer V4.1-Flash architecture.

3. The Complete Model Map

DeepSeek-V4.1-Flash is built around a 40-layer causal language backbone partitioned into a 20-layer causal encoder (layers 0–19) and a 20-layer causal decoder (layers 20–39). Outside this main trunk, a 32-layer vision encoder (DeepSeek-ViT) projects images into the token stream, two Engram modules inject n-gram parameter memory at layers 1 and 14, and a 3-layer DSpark module drafts speculative tokens for fast decoding.

Despite the "Flash" moniker, the model is not a small dense network: it carries 552B parameters in its language and vision backbone plus 196B parameters in Engram tables (748B total). Its speed and small memory footprint come from activating only 8B to 16B parameters per token (during bulk prefill and full-backbone decode, respectively) and aggressively sharing KV caches across layers.

Architectural specifications

Layer-by-layer attention schedule

Every layer in the 40-layer backbone maintains its own 128-token sliding-window attention (SWA) cache. For long-range context, layers 0 and 1 use SWA exclusively, while layers 2 through 39 use Compressed Sparse Attention v2 (CSA2) organized into shared groups:

• Encoder layers 2–19 compress adjacent token pairs with ratio $r = 2$ (one global KV entry per two sequence positions). These 18 layers are divided into three 6-layer groups: layers 2–7, 8–13, and 14–19. The first layer of each group (layers 2, 8, and 14) runs in Full mode, projecting a new shared global KV bank and running its indexer to select top-$K$ entries. The remaining five layers in each group run in Reuse mode, reusing both the group's KV bank and its selected token indices.
• Decoder layers 20–39 operate at compression ratio $r = 1$ (one global KV entry per sequence position) and share a single global KV bank projected from the final encoder output (layer 19) at layer 20. These 20 layers are divided into five 4-layer groups. Layer 20 runs in Full mode (creating the shared decoder KV bank and computing the initial selection plus the 16,384-entry candidate pool). Layers 24, 28, 32, and 36 run in Reindex mode—they read from layer 20's shared KV bank, but run their own indexers over the candidate pool to refresh the 512 selected indices. The remaining three layers in every decoder group run in Reuse mode.

Across the entire 40-layer model, there are only 4 global KV banks (three at ratio 2 in the encoder, one at ratio 1 in the decoder) and 8 index-computing layers (four Full, four Reindex). Out of the 38 CSA2 layers, 30 layers operate in Reuse mode, performing neither global KV projection nor index search. These exact mappings are encoded in the kv_source_layer and index_source_layer arrays in the released config.json.

Distinguishing KV banks, queries, and index selections

Because V4.1-Flash shares state across layers, it helps to keep three attention objects distinct:

1. Global KV bank: The cached 512-channel compressed latents (and derived indexer keys) representing historical positions.
2. Selection indices: The list of up to 512 historical positions chosen for a given query token to attend to.
3. Layer query ($q_t$): The query tensor projected from the current layer's hidden state.

Even in a Reuse layer—where both the global KV bank and the 512 selected indices are inherited from an earlier layer—the layer still projects its own fresh 64-head query $q_t$ from its current hidden state, computes fresh attention weights over those 512 global entries plus its own private 128-token local window, applies its own group output projections, and routes the result through its own 384-expert MoE block. Sharing the KV bank and index list eliminates redundant memory storage and index scans without collapsing the 20-layer decoder into a static lookup.

Why does the decoder keep compression ratio $r = 1$ when the encoder uses $r = 2$? Because all 20 decoder layers share a single global KV bank, leaving that bank uncompressed along the sequence axis costs the same memory as just two $r = 2$ encoder banks while preserving single-token resolution for every decoder query and Reindex step.

How the Causal Encoder-Decoder (CED) changes prefill

In a conventional Transformer, layer $l$ constructs its KV cache from layer $l$'s own input hidden states, forcing every prompt token to pass sequentially through all 40 layers during prefill. In V4.1-Flash's CED architecture, the shared decoder KV bank (anchored at layer 20) is projected directly from the final encoder output (the hidden state exiting layer 19) rather than from deeper decoder activations.

As a result, when ingesting a long prompt, the serving engine only needs to run the 20 encoder layers and the layer-20 global KV projection across the bulk of the prompt. The 20 decoder layers are skipped for all prefix positions except a 128-token suffix (plus the final token), which is replayed through the decoder to populate the 128-token local SWA windows and produce the first output token's logits. During autoregressive decoding, each newly generated token passes through all 40 layers as usual. (Note that the minimal public reference script inference/model.py runs a straightforward 40-layer loop in Transformer.forward rather than implementing the production prefill skip; see Chapter 26.)

Auxiliary modules: Engram and DSpark

Two specialized modules sit alongside the attention-and-MoE backbone:

• Engram (conditional parameter memory): At layers 1 and 14, the model hashes short token n-grams from the input sequence to look up learned embedding vectors from a 196B-parameter table, gates them against the current hidden state, and adds them to the residual stream. Unlike the KV cache—which stores dynamic activations from the current session—Engram stores static, learned lookup tables that offload rote lexical and factual patterns from the MoE experts.
• DSpark (speculative decoding): Rather than training multi-token prediction (MTP) heads jointly during the 45T-token pre-training run, V4.1-Flash trains a lightweight 3-block draft module (DSpark) after backbone pre-training with the backbone weights frozen. DSpark uses a 128-token sliding window to propose 5 draft tokens per step, which the main backbone verifies in parallel. (In the released config.json, these draft blocks retain legacy nextn field naming from earlier DeepSeek releases.)

Summary of changes from DeepSeek-V4-Flash

Compared with DeepSeek-V4-Flash, V4.1-Flash replaces the hybrid of CSA and heavy sequence compression with a unified CSA2 schedule (after two SWA-only layers), removes overlapping compression windows and absolute positional embeddings in the compressor, projects indexer keys directly from the compressed main KV latents, introduces cross-layer KV and index sharing plus hierarchical decoder candidate pools, and adds native vision (DeepSeek-ViT), CED, Engram, Single-Pass mHC, FP4 KV quantization-aware training, and DSpark.

These features are introduced at different points in the model's training and deployment lifecycle:

1. Vision pre-preparation: DeepSeek-ViT is pre-trained separately before joining the backbone.
2. Main multimodal pre-training (45T tokens): Trains the 40-layer CED backbone, CSA2 sharing schedule, Single-Pass mHC, Engram, and vision integration.
3. DSpark training: Freezes the backbone and trains the 3-block speculative drafter.
4. Post-training and adaptation (SFT, RL, OPD): Adapts the model to hierarchical sparse indexing, FP4 main-cache quantization, and bounded decoder local-state replay while training agentic tool use and controllable reasoning effort.
5. Production serving: Pairs the trained model with tiered HBM/DRAM/SSD cache management, Encoder-Prefill-Decode (EPD) disaggregation, and fused kernels.

Sources: Report Figure 3 and §§2.1–2.4 (pp. 7–14), §4.2.1 (pp. 21–22), and released config.json (text_config and vision_config). Chapter 4 breaks down the exact parameter accounting across the MoE layers, Engram tables, and vision encoder.

4. Experts and Parameter Accounting

Each of the 40 backbone layers in DeepSeek-V4.1-Flash contains a Mixture-of-Experts (MoE) feed-forward block with 1 shared expert and 384 routed experts, of which $k = 6$ routed experts are selected per token. Understanding how a 552B-parameter backbone executes with only 8B active parameters during prefill and 16B during decode requires separating three quantities: total weight capacity, active per-token weights, and cluster memory footprint.

Inside a single SwiGLU expert

Each expert is a gated SwiGLU feed-forward network with model hidden dimension $d = 5{,}120$ and intermediate dimension $f = 2{,}304$:

With three projection matrices ($W_{\mathrm{gate}}, W_{\mathrm{up}} \in \mathbb{R}^{f \times d}$ and $W_{\mathrm{down}} \in \mathbb{R}^{d \times f}$), each expert holds $3df$ weights:

To prevent numerical spikes in deep residual pathways, V4.1-Flash also clamps the intermediate expert activation at a magnitude threshold of 10 before applying $W_{\mathrm{down}}$.

Given an input hidden state $h$, the MoE block combines the deterministic output of the shared expert with a gated sum over the six highest-scoring routed experts $S(h)$:

In the released configuration (config.json), router logits pass through a sqrt_softplus scoring function, the top-6 selected affinities are normalized to sum to 1, and the weighted sum is multiplied by a routed_scaling_factor of $1.5$.

Reconciling 552B total parameters with 8B/16B active parameters

Summing all 385 experts (1 shared + 384 routed) across all 40 backbone layers accounts for nearly the entire 552B backbone parameter count:

The remaining $\sim 7\text{B}$ parameters reside in token embeddings, attention projections, indexers, mHC stream-mixing parameters, and final output heads.

For a single token traversing all 40 layers during decoding, 7 experts execute per layer (1 shared + 6 routed), which comes to:

Adding the non-expert attention, embedding, and output projections brings the full-backbone per-token active count to the reported 16B active parameters during decode. During bulk prefill, the Causal Encoder-Decoder (CED) runs only the 20 encoder layers (plus the shared decoder KV projection), cutting the active parameter path in half to the reported 8B active parameters per prefill token.

Notice why simply multiplying $552\text{B} \times (6/384) \approx 8.6\text{B}$ gives the wrong answer: the $6/384$ sparsity ratio applies only to the routed expert pool, whereas shared experts and attention projections execute on every token at every active layer.

Batch-level expert activation and serving hardware

While a single token touches only 6 of the 384 routed experts per layer, a batch of hundreds of concurrent tokens distributed across different contexts will typically activate nearly all 384 experts. During decoding, this creates an arithmetic intensity bottleneck: if a node holds all 384 experts in HBM, it must read hundreds of gigabytes of expert weights from memory just to multiply each expert matrix by the handful of tokens routed to it.

Production deployments therefore distribute experts across multiple GPUs using expert parallelism (EP), using all-to-all collective communication to dispatch token hidden states to the GPUs hosting their selected experts and gather the results back. Consequently, an 8B/16B active parameter count reduces per-token FLOPs, but the serving cluster still needs enough aggregate HBM to hold the full 552B backbone plus the 196B Engram parameter tables, and enough interconnect bandwidth to sustain all-to-all expert dispatch.

Modality-specific load balancing

If an MoE router collapses onto a small subset of experts, those overloaded GPUs become stragglers in expert-parallel execution while underutilized experts waste parameter capacity. Following DeepSeek-V3, V4.1-Flash uses auxiliary-loss-free load balancing as its primary mechanism: each expert $e$ maintains a learned selection bias $b_e$ updated at a rate of $0.001$ based on recent token load. The router selects the top-6 experts using the biased scores $s_e(h) + b_e$, but computes the gating weights $g_e(h)$ from the unbiased scores $s_e(h)$ so that load-balancing adjustments do not distort the forward pass gradients. A small sequence-level auxiliary balancing loss with coefficient $10^{-4}$ is also retained to prevent severe routing skew within any single sequence.

In a native multimodal model, however, tracking only the combined token load across text and vision tokens can mask severe modality collapse. Because visual patch embeddings and text tokens have very different activation distributions and batch proportions, an expert could receive an average overall load by processing exclusively text tokens while visual tokens pile onto a tiny fraction of experts.

To prevent this, V4.1-Flash maintains two separate sets of expert load-balancing biases: one bias vector $\mathbf{b}^{\mathrm{text}} \in \mathbb{R}^{384}$ updated strictly from text-token routing counts, and a second bias vector $\mathbf{b}^{\mathrm{image}} \in \mathbb{R}^{384}$ updated strictly from visual-token routing counts (both with bias update speed $0.001$). Each token uses its own modality's bias vector during top-6 selection, ensuring that both text and image representations spread evenly across the 384 experts in every layer while still sharing the underlying expert weights.

Accounting for Engram and deployment memory

Alongside the 552B backbone parameters, the report allocates 196B parameters to the two Engram conditional-memory modules at layers 1 and 14, bringing the combined backbone-plus-Engram total to 748B parameters (excluding the 32-layer DeepSeek-ViT encoder and 3-layer DSpark drafter).

Unlike MoE weights—which require matrix multiplications—Engram tables act as large key-value embedding stores indexed by hashed token n-grams. Because Engram indices depend only on input token IDs rather than intermediate hidden states, the serving system can place the 196B Engram tables in host DRAM and prefetch the required rows asynchronously over PCIe/CXL before the token reaches layers 1 and 14.

When sizing a deployment cluster, three memory pools scale independently:

1. Backbone weights (552B): Resident in GPU HBM across the tensor/expert-parallel group (roughly 552 GB at 8-bit/1-byte average precision, or 276 GB at 4-bit/0.5-byte average precision, before scales and metadata).
2. Engram tables (196B): Resident in HBM or offloaded to host DRAM with deterministic prefetching.
3. KV caches (890 bytes/token global + local SWA): Scales linearly with active sequence length and batch concurrency across HBM, host DRAM, and SSD tiers.

Sources: Report §2.1.1 (p. 8), §4.2 (pp. 21–22), and released config.json. Chapter 5 traces how raw images are encoded into visual tokens before entering this MoE backbone.

5. Images Inside the Language Model

DeepSeek-V4.1-Flash is a native multimodal model for image-and-text understanding with text generation. Instead of bolting a vision adapter onto a frozen text model after pre-training, V4.1-Flash trains on interleaved image and text sequences throughout its main 45T-token pre-training run.

From pixels to patches: DeepSeek-ViT

Visual inputs are processed by DeepSeek-ViT, a 32-layer Vision Transformer with hidden width $1{,}024$, 16 attention heads, and a $14\times 14$ pixel patch size. An input image is partitioned into non-overlapping $14\times 14$ patches, and each patch's raw pixel values are flattened and mapped into the $1{,}024$-dimensional embedding space.

Standard Vision Transformers typically implement patch extraction as a 2D strided convolution (Conv2d). DeepSeek-ViT replaces that convolution with a pure linear patch projection. Reshaping patches into a 2D matrix multiplication makes the input projection directly compatible with the Muon optimizer, which orthogonalizes 2D weight updates across the network (see Chapter 16).

To handle variable image resolutions and aspect ratios without interpolating fixed absolute positional embedding tables, DeepSeek-ViT applies two-dimensional rotary position embeddings (2D-RoPE) inside its self-attention layers, encoding both vertical (row) and horizontal (column) patch coordinates directly into query-key dot products.

Token compression via $3\times 3$ pixel-unshuffle

Feeding raw $14\times 14$ patch grids directly into a 1M-context language model would rapidly exhaust the token budget. For example, a $1{,}344\times 1{,}344$ image divides into $(1344/14)\times(1344/14) = 96\times 96 = 9{,}216$ patches.

To compress the spatial sequence length by $9\times$ before entering the language backbone, V4.1-Flash applies a $3\times 3$ pixel-unshuffle (space-to-depth) operation to the final DeepSeek-ViT feature map. Each contiguous $3\times 3$ block of patch vectors (each of width $1{,}024$) is concatenated along the channel axis into a single vector of width $9\times 1{,}024 = 9{,}216$. In our $1{,}344\times 1{,}344$ example, the $96\times 96$ patch grid folds into a $32\times 32 = 1{,}024$-token grid of 9,216-dimensional vectors.

A two-layer MLP vision-language projector then maps each 9,216-dimensional vector down to the language backbone's hidden dimension of $5{,}120$. Because pixel-unshuffle simply packs nine neighboring patch features into channels before the learned MLP projection, fine spatial features can be combined by the MLP rather than being discarded by naive spatial pooling.

In the released preprocessing pipeline (inference/image_processor.py), projected visual patch grids are wrapped with special protocol tokens—including image-start, image-end, and per-row newline delimiters—so the total sequence positions consumed by an image equals its downsampled patch count plus these structural markers.

Two-stage vision encoder preparation

Before joining the 552B language backbone for the main 45T-token pre-training run, DeepSeek-ViT is pre-trained in two dedicated stages:

1. Stage 1: Contrastive pre-training (SigLIP). The vision encoder is trained on roughly 47B image-text pairs using the sigmoid contrastive loss (SigLIP). Images in this stage are capped at a maximum resolution of roughly $224\times 224$ (downscaling larger images while preserving aspect ratio). The authors note that training the contrastive stage at higher resolutions substantially increased compute cost while yielding minimal downstream benefit once Stage 2 was applied.
2. Stage 2: Generative alignment with a small proxy MoE. DeepSeek-ViT is attached to a lightweight 4B-parameter MoE language model and trained with an autoregressive next-token prediction objective on 236B tokens of dense captions, alt-text, charts, and document OCR data. During this stage, out-of-bound image resolutions are scaled proportionally into the range between $544\times 544$ and $1{,}344\times 1{,}344$, forcing the encoder and 2D-RoPE representations to preserve fine-grained text and chart details.

Once Stage 2 completes, the temporary 4B MoE language model is discarded, and the prepared DeepSeek-ViT weights are transferred into the main V4.1-Flash pre-training pipeline.

Joint multimodal pre-training schedule

During the main 45T-token pre-training run, the 40-layer language backbone ingests both text and visual tokens from the beginning. To maintain training stability and throughput early in the run, the bulk of DeepSeek-ViT's parameters are initially kept frozen, leaving only the vision encoder's final normalization layer and the two-layer MLP projector trainable alongside the language backbone.

When the pre-training schedule enters its learning-rate decay phase, the entire DeepSeek-ViT encoder is unfrozen and updated jointly with the backbone using a reduced vision learning rate. Coupled with the modality-specific MoE load-balancing biases covered in Chapter 4, this schedule allows the language backbone and vision encoder to co-adapt without early gradient instability.

Visual tokens in agent workflows

In agentic tasks—such as frontend web development, UI verification, chart analysis, or visual search—the model inspects screenshots or rendered plots, emits reasoning traces and tool calls (e.g., cropping, zooming, or running Python scripts to inspect image regions), and incorporates the tool outputs into subsequent turns. Because visual tokens enter the same causal encoder-decoder and CSA2 compression pipeline as text tokens, pairs of visual tokens in encoder layers 2–19 are compressed at ratio $r = 2$ into the encoder global KV banks while remaining at ratio $r = 1$ in the shared decoder global bank.

Sources: Report §2.1.1 (p. 8), §2.5 (p. 15), §§4.1–4.2 (pp. 20–23), and released config.json (vision_config) and inference/image_processor.py. Chapter 6 examines how the Causal Encoder-Decoder processes these multimodal token sequences at half the prefill depth of a standard Transformer.

6. Causal Encoder-Decoder: Cheaper Prompt Processing

In a standard $L$-layer causal Transformer, layer $\ell$ constructs its key-value cache from layer $\ell$'s own input hidden states. Consequently, even during prompt prefill—where the model does not need to emit output tokens for any prefix position before the final token—every prompt token must still pass sequentially through all $L$ layers just to populate the KV caches of the upper layers for future decoding steps.

DeepSeek-V4.1-Flash breaks this dependency with its Causal Encoder-Decoder (CED) architecture. Instead of projecting each decoder layer's global KV cache from that decoder layer's own hidden states, CED projects all decoder global KV entries directly from the final causal encoder output (the hidden state exiting the 20th layer).

Breaking the sequential depth dependency

Let the backbone have $L = 40$ layers, where the first $L/2 = 20$ layers (zero-indexed layers 0–19) form the causal encoder and the remaining $L/2 = 20$ layers (zero-indexed layers 20–39) form the causal decoder. Denote the hidden state exiting the final encoder layer by $H_{L/2}$ (using 1-indexed notation from the report's equations, corresponding to zero-indexed layer 19).

In CED, the global KV representation $C^\ell$ and compressor gating logits $Z^\ell$ for any decoder layer $\ell > L/2$ are projected directly from $H_{L/2}$:

In V4.1-Flash's specific layer schedule (Chapter 3), all 20 decoder layers share a single global KV bank at compression ratio $r = 1$ (no token pooling, so $Z^\ell$ is unused in the decoder), created at the first decoder layer (zero-indexed layer 20). Thus, once the 20-layer causal encoder has computed $H_{L/2}$ for a prompt prefix, a single linear projection $H_{L/2} W_{KV}^{20}$ immediately populates the global KV bank used by all 20 decoder layers.

Why call the lower stack a "causal encoder" rather than a traditional sequence-to-sequence encoder like T5 or BART? A traditional encoder-decoder runs bidirectional self-attention over a fixed input prompt and uses separate cross-attention layers in the decoder. V4.1-Flash's encoder is strictly causal (every position attends only to past positions), and there is no separate cross-attention block: during autoregressive decoding, newly generated tokens pass through both the 20 encoder layers and the 20 decoder layers in a unified causal sequence.

The sliding-window exception and Decoder Bounded Replay

If the decoder only ever read from the global KV bank $C^{20}$, prefill could skip the 20 decoder layers on every prompt position except the very last token (which must traverse layers 20–39 to produce the first output token's logits).

However, every decoder layer $\ell \in \{20, \dots, 39\}$ also maintains its own private 128-token sliding-window attention (SWA) cache computed from that layer's own hidden states $H_{\ell-1}$. Because each layer's SWA window looks back $n_{\mathrm{win}} = 128$ positions into the previous layer's hidden states, computing the exact SWA cache at the final decoder layer would theoretically require propagating a receptive field of up to $20 \times 128 = 2{,}560$ trailing prompt tokens through the decoder stack.

To avoid a 2,560-token decoder tail pass, V4.1-Flash uses Decoder SWA Bounded Replay: during prefill, only the final $n_{\mathrm{win}} = 128$ prompt positions are passed through the 20 decoder layers, with each decoder layer's local attention restricted to that 128-token segment (plus any global KV entries retrieved from the full prompt via $C^{20}$).

For a prompt of length $N$, this reduces bulk prefill complexity from

When $N \gg n_{\mathrm{win}}$ (for example, $N = 1{,}000{,}000$ and $n_{\mathrm{win}} = 128$), instead of executing $40{,}000{,}000$ layer-token transformations, the system executes $20{,}000{,}000$ encoder layer-token transformations plus $128 \times 20 = 2{,}560$ decoder replay transformations—cutting bulk prefill compute and active expert parameters in half (from 16B to 8B active parameters per prefill token). For short prompts where $N \le 128$, the entire prompt falls inside the replay window and passes through all 40 layers as usual.

Prefill and prefix-cache workflows under CED

CED interacts with prefix caching and local-state reconstruction at two boundaries (analyzed in detail in Chapter 11):

1. Encoder boundary on a prefix-cache hit: Persistent prefix caches store only the 4 global KV banks, omitting the 40 per-layer SWA caches to save storage. When a cached prefix is loaded and new input tokens arrive, the encoder first runs a 128-token Encoder SWA Bounded Replay on the tail of the cached prefix to warm up the encoder's local SWA caches, and then runs the 20 encoder layers across the new input suffix.
2. Decoder boundary at the end of prefill: Once the encoder finishes processing the new suffix and projects the new entries into the shared decoder global KV bank at layer 20, the decoder runs Decoder SWA Bounded Replay over only the last 128 positions of the prompt to populate the decoder SWA caches and emit the first predicted token.

Trade-offs and reference code implementation

Projecting the decoder's global KV cache from layer 19's output $H_{L/2}$ means the keys and values stored in the decoder's global bank have undergone 20 layers of contextualization rather than 21 to 40 layers. While each decoder layer still forms its own deep query $q_t^\ell$ from its evolving hidden state $H_{\ell-1}$, re-indexes into the candidate pool every four layers, and applies its own MoE experts and local SWA window, the distant global memory it reads is fixed at the encoder-decoder boundary.

In the public repository's minimal reference implementation (inference/model.py), Transformer.forward simply iterates through all 40 layers sequentially without implementing the encoder-only bulk prefill skip or the cross-layer $H_{L/2}$ wire-up for layer 20 (Attention._compress_kv reads its input x directly). Reproducing the $2\times$ prefill speedup in a custom serving engine requires wiring layer 20's compressor to the output of layer 19 and restricting layers 20–39 to the trailing 128-token window during prefill.

Sources: Report §2.2 (p. 9), §3.2.2 (p. 20), and reference inference/model.py (Transformer.forward, Attention._compress_kv, Compressor.forward). Chapter 7 looks inside the CSA2 compressor to see how token pairs and latents are formed and stored.

7. What Attention Compression Stores

Standard multi-head attention stores separate key and value vectors for every attention head at every sequence position, which quickly blows up HBM capacity at million-token contexts. DeepSeek-V4.1-Flash's Compressed Sparse Attention v2 (CSA2) compresses this state in three ways: it stores a single shared 512-channel latent vector across all 64 query heads, pools pairs of adjacent positions in the encoder ($r = 2$) while keeping single-position granularity in the decoder ($r = 1$), and projects its lightweight 128-channel indexer keys directly from the compressed main latent.

The shared 512-channel main KV latent

Instead of caching 64 separate key and value heads, CSA2 stores one 512-dimensional vector per global cache entry, split into a 448-channel content sub-vector and a 64-channel decoupled rotary position embedding (RoPE) sub-vector (qk_rope_head_dim = 64 in config.json). Both components are quantized to FP4 in the global cache.

Sharing a single 512-channel latent across all 64 query heads does not force the 64 heads to compute identical attention patterns. Each head $h \in \{1, \dots, 64\}$ projects its own distinct 512-dimensional query vector $q_{t,h}$ from the layer's current hidden state (via a 1,280-dimensional low-rank query compression bottleneck), computes its own dot-product alignment scores against the shared 512-channel latents, and routes the resulting weighted latent sum through 8 grouped output projections (kv_lora_rank / group intermediate width $1{,}024$ per 8-head group).

Learned non-overlapping pooling in CSA2 ($r = 2$)

In DeepSeek-V4-Flash, the original CSA compressor pooled overlapping windows of size $2m$ for compression ratio $m$ and added learned absolute positional embeddings inside the compressor. CSA2 simplifies this design by switching to non-overlapping contiguous blocks of size $r$ and dropping the internal absolute positional embedding.

In the three encoder global KV banks (layers 2, 8, and 14), where $r = 2$, each non-overlapping pair of token positions $\mathcal{B}_b = \{2b, 2b+1\}$ is merged into a single 512-channel latent vector. As implemented in the reference Compressor class (inference/model.py), for each position $j \in \mathcal{B}_b$ and channel $c \in \{1, \dots, 512\}$, the model projects a candidate latent vector $u_j = h_j W_{KV}$ and a per-channel importance logit vector $z_j = h_j W_Z$, normalizes the weights across the block via a channel-wise softmax, and takes the weighted sum followed by RMSNorm:

Because the softmax gating weights $\alpha_{j,c}$ are computed independently for every channel $c$, the compressor is not restricted to a uniform average or to picking one token over the other globally. To see how the channel-wise weights operate, suppose a two-token block has two channels with projected values $u_1 = (2, 10)$ and $u_2 = (8, 4)$. If channel 1 assigns softmax weights $(0.75, 0.25)$ while channel 2 assigns $(0.20, 0.80)$, the pooled vector before normalization is $(0.75\times 2 + 0.25\times 8,\; 0.20\times 10 + 0.80\times 4) = (3.5, 5.2)$—preserving channel 1 mainly from the first token and channel 2 mainly from the second. Even with channel-wise gating, folding two 512-dimensional vectors into one is a lossy projection, so the network must learn during pre-training which token features need to survive into long-range global memory.

In the shared decoder global bank at layer 20, the compression ratio is $r = 1$. When $r = 1$, no block pooling or gating matrix $W_Z$ is needed; the compressor simply projects $h_t W_{KV}$ and applies RMSNorm, retaining one 512-channel global entry per sequence position.

Causal visibility and odd-length prefixes

Pooling pairs of tokens introduces a subtle causality constraint during training and decoding. In block $\mathcal{B}_b = \{2b, 2b+1\}$, the pooled latent $\widetilde{c}_b$ mixes information from both token $2b$ and token $2b+1$. If a query at position $2b$ were allowed to attend to $\widetilde{c}_b$, information from future token $2b+1$ would leak backward into position $2b$'s prediction.

To prevent causal leakage, a compressed block $\mathcal{B}_b$ only becomes causally visible to queries at or after its final constituent position ($t \ge 2b+1$). During autoregressive decoding, when the prefix length $N$ is odd, there are $\lfloor N/2 \rfloor$ completed entries in each encoder global bank and one pending uncompleted token $N-1$. The compressor buffers the partial state for that trailing token until the next token arrives to complete the pair. Meanwhile, no information is lost to the current query, because the last 128 tokens are always directly visible at single-token resolution through the layer's local sliding-window attention (SWA) cache.

When applying rotary position embeddings (RoPE) to a compressed block $\mathcal{B}_b$, the reference implementation anchors the block's rotary angle to its first constituent position ($2b$), while the causal mask gates visibility by its last constituent position ($2b+1$). For million-token context support, the compressed global path uses a higher RoPE base frequency (rope_theta) paired with YaRN frequency scaling in config.json.

Projecting indexer keys from the main latent

To select up to $K = 512$ global entries without computing full 64-head attention over the entire history, each Full and Reindex layer uses a lightweight 32-head indexer with head dimension $d_I = 128$.

In V4-Flash, indexer keys were projected separately from the full-width 5,120-dimensional hidden states. In V4.1-Flash's CSA2, the 128-channel indexer key is projected directly from the 512-channel compressed main KV latent (before the main latent undergoes RoPE rotation and FP4 quantization), and then receives its own rotary embedding and FP8 quantization. Deriving the indexer key from the normalized main latent cuts projection parameters and FLOPs while ensuring that the indexer scores a direct linear projection of the exact latent vector that main attention will read.

Complementarity of local SWA and global CSA2

At every CSA2 layer, the attention output combines contributions from up to 512 sparse global entries (stored in FP4 at $r=2$ in the encoder or $r=1$ in the decoder) and up to 128 local sliding-window entries (stored in FP8 at $r=1$ in every layer). The local SWA cache gives every layer unpooled, higher-precision access to the immediate 128-token neighborhood (including any token waiting to complete an $r=2$ encoder pair), while the global CSA2 banks supply long-range retrieval across the full million-token context.

Sources: Report §2.3 (pp. 9–10), §4.2.1 (p. 22), and released inference/model.py (Compressor, Indexer.forward, Attention._compress_kv) and config.json. Chapter 8 examines how these compressed KV banks and sparse selection indices are shared across layers.

8. Sharing Memory and Sparse Selections Across Layers

CSA2 decouples creating a global KV bank from selecting which entries in that bank to attend to. Every CSA2 layer (layers 2–39) is statically assigned one of three operating modes—Full, Reindex, or Reuse—allowing multiple layers to share a single KV bank while refreshing their sparse retrieval indices only at designated group boundaries.

The three CSA2 layer modes

• Full layers (layers 2, 8, 14 in the encoder; layer 20 in the decoder) project a new global KV bank, project the corresponding 128-channel indexer keys, run the 32-head indexer over the bank, and emit the top-$512$ selected indices (plus the 16,384-entry candidate pool at layer 20).
• Reindex layers (layers 24, 28, 32, 36 in the decoder) reuse the active global KV bank and indexer keys from layer 20, but project a fresh indexer query $q^I$ from their own current hidden state to select a new set of top-$512$ indices from the candidate pool.
• Reuse layers (the remaining 30 CSA2 layers) reuse both the active global KV bank and the most recent top-$512$ index list from their group's Full or Reindex leader.

Every mode executes a full attention calculation: even a Reuse layer computes its own 64-head main query $q_t$, updates its own 128-token local SWA cache, computes fresh softmax attention weights over the 512 inherited global entries plus 128 local entries, and passes the result through its own grouped output projections and 384-expert MoE block.

Bank sharing versus index sharing

Bank sharing and index sharing operate on different schedules and must remain strictly aligned:

• In the encoder (layers 2–19): Three 6-layer groups each start with a Full layer (2, 8, 14) followed by five Reuse layers. Here, bank sharing and index sharing coincide: each group has its own $r = 2$ compressed KV bank and uses the single index selection computed at the start of the group.
• In the decoder (layers 20–39): All 20 layers share the single $r = 1$ global KV bank created at layer 20, while five 4-layer groups refresh the top-512 index selection at layers 20 (Full) and 24, 28, 32, 36 (Reindex).

Because encoder banks are compressed at ratio $r = 2$ (where index $b$ addresses token pair $\{2b, 2b+1\}$) whereas the decoder bank is uncompressed at $r = 1$ (where index $j$ addresses token $j$), an index list is only valid for the KV bank it was scored against. In both the reference code (config.json arrays kv_source_layer and index_source_layer) and the distributed training runtime, every Reuse layer is bound to the exact Full/Reindex source of its group.

Quantifying the storage and indexing reductions

For a sequence of length $N$, the three $r = 2$ encoder banks each hold $N/2$ global entries and the single $r = 1$ decoder bank holds $N$ global entries, giving a total of

across the entire 40-layer model (ignoring any trailing odd token waiting to complete an $r = 2$ pair). If all 38 CSA2 layers instead maintained independent global KV banks at those same compression ratios, the model would store $18(N/2) + 20N = 29N$ global entries—an $11.6\times$ reduction from cross-layer bank sharing alone. Compared against DeepSeek-V4-Flash (which used a different hybrid layer schedule and precision format), the report states that V4.1-Flash's global KV cache footprint is roughly one-quarter the size.

On the compute side, only 8 of the 38 CSA2 layers (4 Full, 4 Reindex) run an indexer search, eliminating indexer query projections, key scans, and top-$K$ selection kernels from the other 30 layers. Note that index reuse occurs across depth for the same token position, not across time steps: when decoding token $t+1$, the 8 source indexers still run for position $t+1$ so that every newly generated token selects context suited to its own representation.

Expressivity and distributed training implications

Sharing a single global KV bank across all 20 decoder layers means deeper decoder layers cannot rewrite historical global representations; they can only re-query the layer-20 bank (with fresh main queries in every layer and refreshed top-512 index selections every four layers) while evolving their private 128-token SWA caches and hidden states.

During distributed pipeline-parallel training, cross-layer sharing also introduces cross-stage state dependencies: when a KV bank created on one pipeline stage is read by Reuse or Reindex layers on downstream pipeline stages, forward activations must be forwarded along the pipeline and gradients from all consumer layers must accumulate back into the source compressor and indexer weights. Chapter 18 details how the training system handles this with microbatch state tracking and replicated "shadow indexers."

Sources: Report §2.3.1 and Figure 4 (pp. 10–11), §4.2.1 (p. 22), and released config.json (kv_source_layer, index_source_layer). Chapter 9 examines how the four decoder Reindex layers avoid scanning the full million-token history by searching a hierarchical candidate pool.

9. Hierarchical Sparse Indexing

Even when main attention reads only $K = 512$ global entries per query, a naive sparse indexer still has to compute dot products against all $N$ historical keys to pick those 512 entries. In the decoder—where the shared global KV bank is uncompressed ($r = 1$) and holds up to $1{,}000{,}000$ entries—running five full-context index scans per token at layers 20, 24, 28, 32, and 36 would make index search a major decoding bottleneck.

DeepSeek-V4.1-Flash avoids repeated million-token scans through hierarchical sparse indexing: the first decoder layer (layer 20, Full) scans all $N$ visible entries once to select both its own top-512 entries and a coarse candidate pool of up to 16,384 positions, and the four subsequent Reindex layers (layers 24, 28, 32, 36) restrict their searches to that candidate pool.

The 32-head ReLU indexer

In each of the 8 index-producing layers, the indexer projects $H_I = 32$ query heads $q_{t,h}^I \in \mathbb{R}^{128}$ and head-weighting scalars $w_{t,h}$ from the current hidden state, and compares them against the shared 128-channel indexer keys $k_j^I \in \mathbb{R}^{128}$ (projected from the 512-channel main KV latents):

with rotary position embeddings and quantization scales folded into the query-key vectors. Using a ReLU activation summed over weighted heads (rather than a per-head softmax) allows the indexer to evaluate alignment scores in low precision (MXFP4/FP8) and rank candidate positions by a single scalar score $s_{tj}$ per position $j$. Once the top-$512$ indices are chosen, the 64-head main attention module computes its standard scaled dot-product softmax over the corresponding 512-channel main KV latents.

Block-max pooling and the 16,384-entry candidate pool

At decoder layer 20 (Full), the indexer scores all $N$ causally visible entries in the shared decoder KV bank and performs two selections from those scores $s_{tj}$:

1. Layer 20's immediate read set: The top $K = 512$ individual positions with the highest scores $s_{tj}$, used by main attention in layers 20–23.
2. The decoder candidate pool: The visible sequence is partitioned into contiguous blocks of 8 positions. Each 8-position block is scored by the maximum score among its constituent positions: $$S_{t,B} = \max_{j \in \text{block } B} s_{tj}.$$ The indexer selects up to $2{,}048$ blocks with the highest block-max scores $S_{t,B}$. All 8 positions inside each chosen block are retained, forming a candidate pool of at most $$P = 2{,}048 \times 8 = 16{,}384\text{ positions}.$$

When the query reaches the decoder Reindex layers (layers 24, 28, 32, and 36), each Reindex layer projects its own fresh 32-head indexer query $q_{t,h}^I$, evaluates $s_{tj}$ only over the $P \le 16{,}384$ positions in the candidate pool, and picks its own top $K = 512$ entries for main attention.

Grouping the candidate pool into 8-token blocks around high-scoring tokens preserves local neighborhood context: if layer 20's indexer latches onto a variable name or a table header, the surrounding 7 tokens in that 8-token block (such as the variable's value or adjacent cells) automatically enter the candidate pool, allowing deeper Reindex layers with more evolved semantic queries to attend to those neighboring tokens even if their individual scores at layer 20 were modest.

Arithmetic savings at 1M context

Across a visible decoder history of $N$ positions, running five unrestricted decoder indexers would score $5N$ positions per query token. With hierarchical indexing, layer 20 scores $N$ positions and the four Reindex layers each score at most $P = 16{,}384$ positions, reducing total decoder index scoring from

At $N = 1{,}000{,}000$, this drops the number of scored decoder positions from $5{,}000{,}000$ to $1{,}000{,}000 + 4(16{,}384) = 1{,}065{,}536$—a $4.69\times$ reduction in decoder index comparisons (while the three encoder Full layers 2, 8, and 14 continue to scan their half-length $N/2$ compressed banks). When $N \le 16{,}384$, the candidate pool covers the entire prefix and hierarchical indexing behaves identically to unrestricted indexing.

Because the remaining full-context scans (at encoder layers 2, 8, 14 and decoder layer 20) run in low-precision 128-channel indexer space while main 64-head attention and MoE expert execution are independent of $N$, single-token decoding FLOPs stay nearly flat across context lengths. In Report Figure 2, precision-weighted single-token decode FLOPs (weighting BF16, FP8, and FP4 operations as $1$, $0.5$, and $0.25$) grow by only ~25% when expanding context length $256\times$ from 4K to 1M tokens.

Reference code semantics versus production kernels

In the released reference code (inference/model.py, Indexer.forward and select_candidate_blocks), the Reindex path computes the dense score matrix across all keys first and then masks out non-candidate positions with $-\infty$ before calling topk. That reference implementation reproduces the exact mathematical output of hierarchical indexing, whereas the production serving engine uses a sparse gather/indexing kernel that only computes dot products for the $\le 16{,}384$ candidate blocks to realize the $N + 4P$ compute and memory-bandwidth reduction.

Adaptation and cascaded recall risk

Hierarchical sparse indexing is enabled during post-training so that the Reindex layers at depths 24, 28, 32, and 36 learn to rank entries within the layer-20 candidate pool. Even with training-aware adaptation, the two-level hierarchy introduces a cascaded recall constraint: if an 8-token block fails to place among the top 2,048 blocks at layer 20, none of the deeper decoder layers can retrieve it. The authors note in §6 that characterizes and mitigating correlated retrieval misses under aggressive sparse selection remains an active area of work.

Sources: Report §2.3.2 and Figure 5 (pp. 11–12), Figure 2 (p. 5), and released inference/model.py (Indexer.forward, select_candidate_blocks). Chapter 10 examines the 4-bit floating-point formats used to store the main KV latents and indexer keys.

10. Four-Bit Attention Memory

To cut KV cache memory footprint and host-to-device transfer bandwidth, DeepSeek-V4.1-Flash quantizes both its 512-channel global main KV latents and its 128-channel indexer keys to 4-bit floating point (FP4), while keeping the 128-token local sliding-window attention (SWA) caches in 8-bit floating point (FP8). Because main attention and index scoring have different hardware execution paths and dynamic-range profiles, V4.1-Flash uses two distinct block-scaled FP4 formats for the two caches.

Block-scaled FP4 formats: Main KV versus Indexer K

In block-scaled quantization, a vector is partitioned into contiguous blocks of channels, and each channel $i$ in block $b$ is stored as a 4-bit payload $q_i$ paired with a shared block scale factor $s_b$:

V4.1-Flash tailors the payload format, block size, and scale format to each tensor's role:

1. Global Main KV (512 channels): Uses E2M1 (1 sign bit, 2 exponent bits, 1 mantissa bit) 4-bit payloads with one E4M3 (8-bit float) scale factor per 16 channels. This matches the local block layout of NVIDIA's NVFP4 format, but omits the second-level FP32 per-tensor global scale. During attention, selected main KV entries are dequantized back to higher precision before multiplying with the 64-head queries.
2. Indexer Keys $k^I$ (128 channels): Uses the OCP-standard MXFP4 format—E2M1 4-bit payloads with one E8M0 (8-bit power-of-two exponent) scale factor per 32 channels. Using standard MXFP4 for indexer keys and queries allows the 32-head indexer to run directly on native hardware FP4 matrix-multiply tensor cores during full-context and candidate-pool scans.

Exact byte accounting per cached entry

Accounting for both the 4-bit payloads and the 8-bit block scales gives the exact byte size of each global cache entry:

• Per 512-channel Main KV entry:
  • Payload: $512 \text{ channels} \times 0.5\text{ bytes} = 256\text{ bytes}$.
  • Scales: $512 / 16 = 32\text{ blocks} \times 1\text{ byte (E4M3)} = 32\text{ bytes}$.
  • Total: $256 + 32 = \mathbf{288\text{ bytes}}$ per main KV entry ($4.5\text{ effective bits per channel}$, a $1.78\times$ reduction compared to an unscaled 512-byte FP8 payload).
• Per 128-channel Indexer Key entry:
  • Payload: $128 \text{ channels} \times 0.5\text{ bytes} = 64\text{ bytes}$.
  • Scales: $128 / 32 = 4\text{ blocks} \times 1\text{ byte (E8M0)} = 4\text{ bytes}$.
  • Total: $64 + 4 = \mathbf{68\text{ bytes}}$ per indexer key entry ($4.25\text{ effective bits per channel}$).

Together, one main KV entry plus its companion indexer key occupy $288 + 68 = \mathbf{356\text{ bytes}}$. Multiplying $356\text{ bytes}$ by the $2.5$ global entries stored per token across the 40-layer model (three $r=2$ encoder banks plus one $r=1$ decoder bank, as derived in Chapter 8) yields $2.5 \times 356 = \mathbf{890\text{ bytes per token}}$—the exact headline global KV cache footprint reported in the paper.

Why the main KV format omits NVFP4's global scale

Standard NVFP4 includes a second-level FP32 tensor-wide scale factor to prevent E4M3 block scales from overflowing or underflowing on activations with wide dynamic range. In V4.1-Flash, however, every 512-channel main KV latent passes through an RMSNorm layer immediately before rotary embedding and quantization.

If a 512-channel vector has unit root-mean-square after normalization and the learned RMSNorm gain parameters have magnitude bounded near $1$, the Euclidean ($\ell_2$) norm of the entire 512-dimensional vector is roughly

and no individual coordinate can exceed the vector's $\ell_2$ norm (in practice, the authors observe maximum channel magnitudes around $10$ during training). Because an E4M3 scale can represent magnitudes up to $448$ and an E2M1 payload reaches magnitude $6$, their product covers magnitudes up to $448 \times 6 = 2{,}688$—more than two orders of magnitude above the $\le 22.6$ activation bound. Omitting the redundant per-tensor scale simplifies incremental per-token cache appends without risking overflow.

Post-RoPE quantization and Quantization-Aware Training (QAT)

Of the 512 main KV channels, 448 carry content projections and 64 carry decoupled rotary position embeddings (RoPE). Because quantization $Q(\cdot)$ is a nonlinear rounding operation while RoPE rotation $R$ is an orthogonal linear transform, quantization and rotation do not commute:

Quantizing the 64 rotary channels before applying RoPE (so that cached keys are rotated after dequantization at read time) gave a slight accuracy edge in ablations, but required applying rotary rotations to all retrieved keys during every decode step. V4.1-Flash instead applies RoPE before quantizing the 512-channel vector into the FP4 cache, trading a tiny accuracy margin to keep the decode path free of extra key rotations.

To recover from the coarse E2M1 quantization grid, V4.1-Flash applies Quantization-Aware Training (QAT) to the main KV cache during post-training, inserting simulated FP4 quantization-dequantization into the forward pass so queries and output projections adapt to the 4-bit reconstruction noise.

Why the 128-token local SWA cache stays in FP8

While global CSA2 banks are quantized to FP4, all 40 per-layer 128-token sliding-window attention (SWA) caches remain in FP8. The report notes that local sliding-window attention is significantly more sensitive to 4-bit quantization noise than global retrieval. Because each SWA cache is capped at $n_{\mathrm{win}} = 128$ tokens regardless of total context length and is never written to persistent prefix storage (thanks to bounded replay), keeping those 128 local entries in FP8 preserves fine-grained local representations at negligible memory cost.

Sources: Report §2.4.4 (p. 14) and released inference/model.py (Attention._compress_kv, Indexer.forward). Chapter 11 examines how V4.1-Flash avoids persisting those 40 FP8 sliding-window caches by reconstructing them on the fly with 128-token bounded replay.

11. Bounded Local-State Replay

Every layer in DeepSeek-V4.1-Flash pairs global sparse attention with a private 128-token sliding-window attention (SWA) cache stored in FP8. Although 128 tokens per layer is small in active HBM during decoding, persisting 40 layers of unshared FP8 SWA state for every historical token position in DRAM or SSD prefix storage would dwarf the 890-byte global KV cache. Similarly, during prefill under the Causal Encoder-Decoder (CED), computing exact local SWA state for the 20 decoder layers would require propagating thousands of trailing tokens through the decoder.

V4.1-Flash solves both problems with Bounded Local-State Replay: instead of storing or exactly recomputing multi-layer SWA chains, the system reconstructs local SWA caches by replaying only the most recent $W = 128$ token positions with local attention truncated to the replay segment boundary.

Why exact SWA reconstruction grows with depth

In a single layer with sliding window size $W = 128$, the hidden state at position $t$ depends on the previous layer's hidden states across $[t - W + 1, t]$. In the next layer, each of those $W$ positions in turn depends on $W$ positions from the layer below it. Across $L$ stacked layers, the exact local receptive field expands linearly with depth to $1 + L(W - 1) \approx LW$ positions:

• Across all $L = 40$ backbone layers, exact local-state reconstruction requires replaying roughly $40 \times 128 = 5{,}120$ trailing positions.
• Across the $L/2 = 20$ decoder layers, exact local-state reconstruction requires replaying roughly $20 \times 128 = 2{,}560$ trailing positions.

Rather than paying a 2,560- or 5,120-token recomputation penalty on every turn, Bounded Replay truncates the replay span to a single window of $W = 128$ positions.

Mechanics of truncated replay at boundary $s$

Suppose bounded replay begins at sequence position $s = N_{\mathrm{prefix}} - W$. For any position $i \in [s, N_{\mathrm{prefix}} - 1]$ inside the 128-token replay segment, local sliding-window attention is restricted to keys within the replay segment itself:

while global CSA2 attention reads freely from all causally visible entries in the cached global KV banks across $[0, i]$.

Because the earliest positions in the replay segment ($i$ near $s$) cannot attend to local SWA keys prior to $s$, their hidden states at layer 1 are missing local predecessors from $[s - W + 1, s - 1]$. As activations propagate upward through subsequent layers, later positions in the replay window attend to those slightly altered intermediate representations. Thus, even though the final position $N_{\mathrm{prefix}} - 1$ sees a full 128-token window $[s, N_{\mathrm{prefix}} - 1]$ at every layer, the reconstructed SWA cache is an approximation to the state that an uninterrupted full-sequence forward pass would have produced.

Two distinct replay paths: Encoder versus Decoder

Bounded replay operates at two different points in the serving pipeline:

1. Encoder SWA Bounded Replay (on a prefix-cache hit): Long-term prefix caches on DRAM/SSD store only the 4 global CSA2 banks (890 bytes/token), discarding all 40 local SWA caches. When a multi-turn request hits a cached prefix of length $N_{\mathrm{prefix}}$ and appends $N_{\mathrm{new}}$ new tokens, the 20-layer causal encoder runs over the last 128 tokens of the cached prefix followed by the $N_{\mathrm{new}}$ new tokens. For the 128 replayed prefix positions, the encoder only populates the encoder SWA caches without overwriting the already-cached global KV entries; for the $N_{\mathrm{new}}$ suffix positions, it generates both SWA state and new global KV entries.
2. Decoder SWA Bounded Replay (at the end of every prefill): After the encoder has processed the prompt (and projected the new global entries into the shared decoder KV bank at layer 20), the 20-layer decoder runs over only the final 128 encoder output vectors $[N - 128, N - 1]$ of the prompt. This populates the 20 decoder SWA caches for subsequent token generation and produces the logits for the first output token at position $N - 1$. Decoder SWA state is discarded as soon as the generation turn finishes.

To trace how both paths work together, suppose an agent session hits a 100,000-token cached prefix in SSD storage and appends a 2,000-token tool output:

• The serving engine stages the 100,000-token global KV banks ($89\text{ MB}$) into HBM.
• The 20-layer encoder processes $128 + 2{,}000 = 2{,}128$ tokens (regenerating encoder SWA on the 128-token prefix tail and appending 2,000 new tokens to the global KV banks, including the layer-20 decoder projection).
• The 20-layer decoder processes only the final $128$ tokens of the 102,000-token sequence to warm up the decoder SWA caches and emit the first response token.

Why 128-token replay works—and its cache-boundary caveat

Why does truncating a theoretical 2,560- or 5,120-token receptive field down to 128 tokens cause negligible accuracy loss in the report's evaluations? First, effective attention weights in local sliding windows decay sharply with distance, so multi-hop local propagation across dozens of layers carries very little signal compared to direct 1-hop local attention over the immediate 128 predecessors. Second, every layer above layer 1 simultaneously attends to the full prefix history via the exact cached global CSA2 banks, so information prior to position $s$ is still directly accessible through sparse global retrieval. Third, V4.1-Flash simulates Decoder SWA Bounded Replay during post-training so the decoder layers adapt to the truncated boundary distribution.

One subtle systems consequence of Encoder SWA Bounded Replay is prefix-boundary dependence: because the new suffix tokens $[N_{\mathrm{prefix}}, N_{\mathrm{prefix}} + N_{\mathrm{new}} - 1]$ attend to the approximately reconstructed encoder SWA states from $[N_{\mathrm{prefix}} - 128, N_{\mathrm{prefix}} - 1]$, the new global KV entries appended for the suffix differ slightly from the entries that would have been computed if the entire $N_{\mathrm{prefix}} + N_{\mathrm{new}}$ sequence had been prefilled in a single uninterrupted pass. As noted in Report §6, characterizing any edge-case sensitivity to these approximate replay boundaries remains an open robustness question.

Sources: Report §2.2 (p. 9), §§3.2.1–3.2.2 (pp. 19–20), and §6 (p. 37). Chapter 12 combines the 890-byte global cache format and bounded replay into the complete HBM/DRAM/SSD serving architecture.

12. Cache Economics and the Serving System

Serving million-token agent sessions at fleet scale requires matching each class of model state to the right memory tier. DeepSeek-V4.1-Flash reduces the active global KV cache to 890 bytes per token (roughly one-quarter of V4-Flash) and cuts long-term persistent prefix storage to roughly one-eighth of V4-Flash by separating the retention lifetime of global CSA2 banks from short-lived sliding-window attention (SWA) caches.

Deriving the 890-byte per-token global cache footprint

Across the 40-layer backbone, V4.1-Flash maintains four global KV banks: three encoder banks at compression ratio $r = 2$ (layers 2, 8, and 14) and one shared decoder bank at ratio $r = 1$ (layer 20). For a sequence of length $N$, these four banks hold $3(N/2) + N = 2.5N$ global entries.

As calculated in Chapter 10, each global entry consists of:

• A 512-channel main KV latent in E2M1 FP4 with 32 E4M3 block scales: $256 + 32 = 288\text{ bytes}$.
• A 128-channel indexer key in MXFP4 with 4 E8M0 block scales: $64 + 4 = 68\text{ bytes}$.

Combining $288 + 68 = 356\text{ bytes}$ per entry across $2.5N$ entries gives:

That is 890 bytes per sequence position for the global KV cache. Across standard context lengths, the nominal global cache size scales linearly:

Accounting for the bounded local SWA caches in HBM

In addition to the 4 global CSA2 banks, every active request in HBM holds 40 per-layer 128-token local SWA caches in FP8. At 512 channels per entry and 1 payload byte per FP8 scalar, the 40 local windows occupy

before scale factors, DSpark draft-window buffers, and page-table metadata. At $N = 1{,}000{,}000$, $2.62\text{ MB}$ is less than $0.3\%$ of the $890\text{ MB}$ global cache; at short contexts like $N = 4{,}096$ ($3.65\text{ MB}$ global), the local SWA windows represent a substantial fraction of active HBM state.

Why V4.1-Flash drops SWA from persistent storage

If the 40 SWA windows occupy only ~2.62 MB per saved turn endpoint, why did they become a storage bottleneck in V4-Flash?

In DeepSeek's V4-Flash production deployment, persistent prefix caches saved both the global KV banks and the 128-token SWA snapshots at the end of every prompt and assistant turn so subsequent turns could resume without recomputing local state. In interactive agent workloads with frequent short turns (tool calls, shell commands, file reads), a single trajectory generates dozens of intermediate turn endpoints. Across a fleet storing millions of multi-turn sessions for days, those per-endpoint SWA snapshots accumulated until SWA consumed nearly half of total persistent KV storage capacity under V4-Flash's production workload.

V4.1-Flash eliminates SWA from long-lived persistent storage entirely, splitting cache retention into three tiers matched to reuse horizons:

1. Persistent Global KV Tier (SSD / distributed storage, $\ge 72\text{-hour}$ TTL): Stores only the 4 global CSA2 banks (890 bytes/token). Global prefixes exhibit long-tail reuse across paused sessions, forked agent branches, and shared system/repository prompts.
2. Ephemeral Encoder SWA Pool (10% of host DRAM per node, minute-scale TTL): Caches the 20 encoder SWA windows in a distributed host-DRAM pool for active multi-turn sessions while an agent awaits an immediate tool response. If the tool returns within the minute-scale TTL, the encoder resumes directly from host DRAM; if the TTL expires, Encoder SWA Bounded Replay (Chapter 11) reconstructs the encoder SWA state by replaying just 128 tokens.
3. Active HBM Tier (Decoder SWA, single-turn lifetime): The 20 decoder SWA windows exist only in GPU HBM during active token generation. They are created at the end of prefill via Decoder SWA Bounded Replay (128 tokens) and discarded immediately when the turn finishes.

This tiered lifetime policy explains how V4.1-Flash achieves an $8\times$ reduction in persistent KV storage compared to V4-Flash on identical workloads: dropping SWA from persistent disk cuts the stored footprint by roughly $2\times$, and shrinking the remaining global KV cache by $4\times$ (from V4-Flash's layout down to 890 bytes/token) compounds to $0.5 \times 0.25 = 0.125$ ($1/8$) of the prior persistent storage volume.

Encoder–Prefill–Decode (EPD) disaggregation and fused kernels

To prevent multimodal encoding, long-context prefill, and latency-sensitive token decoding from interfering with one another on the same GPUs, the production serving stack uses Encoder–Prefill–Decode (EPD) disaggregation—separating Encoder workers (which run DeepSeek-ViT on image inputs), Prefill workers (which run the 20-layer causal encoder and layer-20 global projection on prompts), and Decode workers (which run full 40-layer autoregressive generation with DSpark speculative decoding) into independently scaled worker pools. (Note that in "EPD disaggregation," the "E" refers to the vision encoder service stage, whereas in "CED" it refers to the 20-layer causal encoder half of the language backbone.)

On the kernel execution side, the serving engine fuses small sequential operations—including RoPE-attention projections, gating, Single-Pass Mega-mHC stream mixing, MoE routing, and top-$K$ selection—into composite kernels. With these fusions and cross-layer Reuse, the 30 Reuse layers execute in just 15 CUDA kernels per layer during prefill and 11 kernels per layer during decode.

Sources: Report §3.2 (pp. 18–20), Abstract, and released config.json and inference/model.py. Chapter 13 examines how Single-Pass Manifold-Constrained Hyper-Connections (Mega-mHC) cut residual memory traffic in half to enable those fused kernels.

13. Single-Pass Manifold-Constrained Hyper-Connections

Instead of a single width-$d$ residual pathway, DeepSeek-V4.1-Flash maintains $n = 4$ parallel residual streams per token (each of width $d = 5{,}120$) connected via Manifold-Constrained Hyper-Connections (mHC). While multi-stream residuals expand the bandwidth of the residual highway without widening the expensive attention or MoE sublayers, standard mHC introduces a sequential reduction barrier that forces GPU kernels to read and write the 4-stream residual state twice at every sublayer boundary.

V4.1-Flash removes that barrier with Single-Pass mHC, shifting the input-mixing coefficient by one block so that residual updates, stream mixing, and next-step coefficient prediction fuse into a single memory pass (Mega-mHC), cutting residual HBM traffic in half.

Multi-stream residuals and the Sinkhorn–Knopp constraint

In a standard Transformer block, a sublayer $F_\ell$ (either an attention module or an MoE module) updates a single residual vector via $x_{\ell+1} = x_\ell + F_\ell(x_\ell)$. Under mHC, each token position maintains a matrix of $n = 4$ parallel residual streams $X_\ell \in \mathbb{R}^{n \times d}$.

Before entering sublayer $F_\ell$, the $n$ streams are collapsed into a single width-$d$ vector via a learned $1 \times n$ mixing vector $A_\ell$. While $F_\ell$ runs on that single vector, the $n$ residual streams are mixed among themselves by an $n \times n$ matrix $B_\ell$, and the sublayer output is broadcast back across the $n$ streams via an $n \times 1$ vector $C_\ell$:

Because $F_\ell$ always operates on the mixed single-stream vector $A_\ell X_\ell \in \mathbb{R}^{1 \times d}$, maintaining four residual streams does not multiply attention or MoE FLOPs by four.

To prevent deep compositions of the $n \times n$ residual mixing matrices $B_\ell$ from exploding or vanishing across 80 sublayer boundaries (40 attention + 40 MoE), mHC constrains $B_\ell$ to lie on the manifold of doubly stochastic matrices (non-negative entries whose rows and columns each sum to 1) by running 20 Sinkhorn–Knopp alternating row/column normalization iterations on the predicted matrix inside $\mathcal{H}$. Because every doubly stochastic matrix forms a convex combination of the $n$ streams, repeated residual mixing preserves total stream mass. (Note that the report also uses a Sinkhorn-style algorithm inside the embedding optimizer in Chapter 16; here, the 20-step Sinkhorn–Knopp procedure operates on the $4 \times 4$ activation-mixing matrix $B_\ell$ during the forward pass.)

The two-pass bottleneck in standard mHC

Look closely at the data dependency in the standard mHC equation above: the predictor $\mathcal{H}(X_\ell)$ computes the dynamic mixing coefficients $(A_\ell, B_\ell, C_\ell)$ from the full width-$d$ RMSNorm and linear projection of $X_\ell$.

When finishing sublayer $\ell-1$ and preparing the input $A_\ell X_\ell$ for sublayer $\ell$:

1. A GPU kernel can update $X_\ell = B_{\ell-1} X_{\ell-1} + C_{\ell-1} F_{\ell-1}(\dots)$ tile-by-tile along the hidden dimension $d = 5{,}120$, accumulating the reduction statistics needed by $\mathcal{H}(X_\ell)$ along the way.
2. However, the input-mixing coefficient $A_\ell$ is not known until the reduction across all $5{,}120$ channels of $X_\ell$ finishes.
3. Consequently, the kernel must write the entire $4 \times 5{,}120$ matrix $X_\ell$ out to HBM, finish computing $A_\ell$, and then launch a second pass that reads $X_\ell$ back from HBM to compute the mixed input $A_\ell X_\ell$ (plus pre-normalization).

Accounting for all reads and writes across the two passes (including pre-normalization), the original multi-kernel mHC implementation incurs $(4n + 4)d$ scalar memory transactions per sublayer boundary—or $20d = 102{,}400$ scalars ($204.8\text{ KB}$ in BF16) per token at every sublayer boundary when $n = 4$.

Single-Pass mHC and the Mega-mHC fused kernel

Single-Pass mHC breaks the reduction dependency on $A_\ell$ by shifting the input-mixing vector by one sublayer block, consuming the already-computed coefficient $A_{\ell-1}$ from the previous boundary:

Because $A_{\ell-1}$ is already available in registers/SRAM before $X_\ell$ is even computed, a single fused kernel (Mega-mHC) tiled across the hidden dimension $d$ can:

1. Read a tile of the $n$ streams $X_{\ell-1}$ and the previous sublayer output $F_{\ell-1}$.
2. Compute the updated tile of $X_\ell$ in registers and write $X_\ell$ to HBM.
3. Immediately multiply that in-register tile of $X_\ell$ by $A_{\ell-1}$ (and apply RMSNorm + FP8 quantization) to write the next sublayer's input $A_{\ell-1} X_\ell$.
4. Simultaneously accumulate the tile's norm and projection statistics to produce $(A_\ell, B_\ell, C_\ell)$ at the end of the kernel for the next sublayer boundary.

With Single-Pass mHC, Mega-mHC achieves the theoretical minimum memory traffic of reading $n$ stream vectors plus 1 sublayer output and writing $n$ updated stream vectors plus 1 mixed input:

For $n = 4$ and $d = 5{,}120$, memory traffic drops from $20d$ ($102{,}400$ scalars) in original mHC—or $(3n + 2)d = 14d$ in a partially fused standard mHC kernel—down to $10d = 51{,}200$ scalars ($102.4\text{ KB}$ in BF16), cutting residual memory traffic in half with negligible impact on model perplexity or downstream accuracy.

Sources: Report §2.4.1 (pp. 12–13), §4.2.1 (p. 22), and released inference/model.py (Block residual-mixing methods). Chapter 14 examines how the two Engram modules inject 196B parameters of n-gram lookup memory into these four residual streams.

14. Engram: Conditional Parameter Memory

Large language models spend significant MoE expert capacity memorizing static lexical associations, named entities, and multi-token idioms. DeepSeek-V4.1-Flash offloads much of that rote lookup work from deep neural computation into Engram, a conditional parameter memory system that allocates 196B learned embedding parameters across two modules placed at zero-indexed backbone layers 1 and 14.

Unlike the KV cache—which stores dynamic activations computed from the current conversation—Engram tables are static learned model weights addressed deterministically by input token n-grams.

Multi-head n-gram hashing and parameter accounting

At each text token position $t$, Engram extracts trailing token n-grams of orders $n \in \{2, 3, 4\}$ (after applying a canonicalizing tokenizer compression mapping that merges case/whitespace variants). Because materializing an exact lookup table for all $V^4$ possible 4-grams is infeasible, each n-gram order is hashed into 8 independent hash heads, where each head addresses a table of roughly $16\text{M}$ rows (using distinct prime moduli around $16\times 10^6$ to decorrelate hash collisions across heads).

In the released configuration (config.json), each hash head retrieves a 256-dimensional vector, giving $8 \times 256 = 2{,}048$ channels per n-gram order. Across all three orders ($2, 3, 4$), a single Engram module performs $3 \times 8 = 24$ row lookups per token and concatenates them into a vector of width

Counting the parameters across all 24 tables in a single Engram module gives:

or $196.6\text{B}$ parameters across the two modules at layers 1 and 14 (matching the report's rounded 196B figure). Stored in FP8 (1 byte per parameter before block scales), the two Engram tables occupy roughly $196\text{ GB}$, while a single token position reads only $24 \times 256\text{ bytes} = 6\text{ KB}$ of raw table payload per module ($12\text{ KB}$ across both modules).

Context-aware gating into the 4-stream residual

Because multiple n-grams can collide in a hash bucket and the same lexical phrase can carry different meanings in different contexts, Engram does not blindly add the retrieved n-gram embeddings to the hidden state. Instead, the concatenated 6,144-dimensional lookup vector is linearly projected (in FP8) into a shared value vector $v$ and per-stream key vectors $k$ (one key per mHC residual stream).

Each of the 4 residual streams in $X$ computes a normalized dot-product alignment with its corresponding Engram key $k$, applies a signed square-root scaling and sigmoid activation to form a context-dependent scalar gate $g(X, k) \in (0, 1)$, and injects the gated value into the residual stream:

If a hash collision retrieves an irrelevant row or the surrounding context contradicts the canonical n-gram meaning, the contextual gate $g(X, k)$ suppresses the contribution before it can corrupt the residual stream. Compared to the earlier standalone Engram design, V4.1-Flash drops the short causal convolution layer (finding its accuracy benefit too small to justify the extra inference state and kernel complexity) and masks out visual patch positions (inference/model.py, Engram and NgramHashState) so image tokens neither pollute lexical n-gram windows nor receive text n-gram injections.

Deterministic host-DRAM prefetching at inference

A key systems advantage of n-gram addressing over MoE routing is that Engram table indices depend only on the input token IDs, which are known as soon as a prompt arrives or a token is sampled—before the forward pass even starts.

Placing the two Engram modules at layers 1 and 14 allows the serving engine to store the 196 GB of FP8 Engram tables in host DRAM and prefetch the required 24 rows per token over PCIe/RDMA in the background:

• While layer 0 executes its attention and MoE blocks on GPU, the engine prefetches the layer-1 Engram embeddings from host memory.
• While layers 2 through 13 execute on GPU, the engine prefetches the layer-14 Engram embeddings.

By overlapping deterministic host-to-device DMA transfers with backbone compute, Engram adds 196B parameters of lookup capacity without consuming scarce GPU HBM during standard serving.

Distributed sharding during training and RL rollouts

During pre-training and post-training, the 196B Engram tables are managed differently depending on the workload:

• Pre-training pipeline parallelism: Placing the two modules at layers 1 and 14 balances parameter memory across pipeline stages. The tables are row-sharded across dedicated Engram process groups (with optimizer states further sharded across data-parallel replicas). Before a pipeline stage begins executing a microbatch, all token n-gram indices for the local batch are prefetched in bulk; during the backward pass, sparse embedding gradients are buffered and returned asynchronously to the owning ranks, overlapping communication with DeepSeek-ViT gradient computation.
• Reinforcement learning (RL) rollouts: During large-scale asynchronous RL rollout generation, keeping Engram tables in host DRAM alongside heavy CPU environment tasks created host memory pressure and fragmentation. In the RL rollout cluster, the Engram tables are therefore kept GPU-resident in HBM across the rollout workers.

Sources: Report §2.4.2 (p. 13), §3.1.3 (p. 18), and released inference/model.py (Engram, NgramHashState) and config.json. Chapter 15 examines how DSpark accelerates autoregressive generation via confidence-scheduled speculative decoding.

15. DSpark and Speculative Decoding

Autoregressive decoding is bottlenecked by sequential step dependencies: generating $k$ tokens normally requires $k$ serial passes through all 40 backbone layers. Speculative decoding breaks this serial bottleneck by using a fast draft module to propose multiple candidate future tokens and verifying them in a single parallel pass of the target model.

In DeepSeek-V4.1-Flash, speculative drafting is handled by DSpark, a 3-layer draft module that uses a 128-token local sliding window to propose up to 5 draft tokens per step in a single forward pass, paired with a lightweight Markov dependency head and a confidence-aware verification scheduler.

Semi-autoregressive drafting with a Markov head

Earlier DeepSeek models (such as V3) used Multi-Token Prediction (MTP) modules chained sequentially—where predicting draft offset $+2$ required running a full draft Transformer block on the output of draft offset $+1$.

DSpark replaces sequential draft blocks with a semi-autoregressive architecture:

1. Parallel feature construction (3 draft blocks): Three lightweight Transformer blocks (with 128-token sliding-window attention in DSparkAttention) run once over the current context to produce base hidden representations for 5 future positions simultaneously.
2. Sequential token conditioning (lightweight Markov head): Predicting 5 future tokens completely independently would ignore joint token dependencies (for instance, after "San", offset $+1$ might pick "Francisco" or "Diego", and offset $+2$ must stay consistent with whichever city was chosen at $+1$). DSpark therefore passes the 5 parallel feature vectors through a tiny Markov head that conditions each drafted token $i \in \{1, \dots, 5\}$ on the sampled token at offset $i-1$ without running another full Transformer block.

Conditional confidence and dynamic verification length

Verifying all 5 drafted tokens in the 40-layer backbone is not always optimal. If draft position 1 is very likely to be rejected by the target model, spending backbone compute and batch capacity verifying positions 2 through 5 wastes hardware throughput. Let $T_D$ denote the time to run DSpark, $T_V(q)$ the time for the 40-layer backbone to verify $q \in \{0, \dots, 5\}$ draft positions in parallel, and $\mathbb{E}[A(q)]$ the expected number of accepted tokens (plus the bonus token emitted by the target model). System throughput scales with

To maximize expected throughput dynamically, DSpark includes a confidence head that predicts the conditional acceptance probability $a_j \in [0, 1]$ for each draft position $j \in \{1, \dots, 5\}$, conditioned on positions $1, \dots, j-1$ having been accepted. The probability that the entire draft prefix of length $i$ survives verification is the product of the conditional probabilities:

For example, if the confidence head estimates conditional acceptance probabilities $(a_1, a_2, a_3, a_4, a_5) = (0.9, 0.8, 0.7, 0.6, 0.5)$ for a given step, the cumulative prefix survival probabilities decay as $(0.900, 0.720, 0.504, 0.302, 0.151)$, giving an expected accepted draft length of $\sum_{i=1}^5 S_i \approx 2.58$ tokens.

The production serving scheduler combines these per-request survival probabilities $S_i$ with offline-profiled hardware throughput curves $T_V(q)$ under the current batch load to choose how many draft tokens $q \le 5$ to send to the 40-layer backbone for verification. Because final token acceptance is still decided by exact speculative verification against the 40-layer target model's logits, miscalibration in the confidence head only affects scheduling efficiency, never the output distribution of the model.

Decoupled training lifecycle

Unlike DeepSeek-V3's MTP heads—which were trained jointly with the main backbone throughout pre-training—V4.1-Flash omits DSpark from the 45T-token main pre-training run:

1. Dedicated DSpark training stage: After main pre-training completes, the 40-layer backbone is frozen and the 3-block DSpark module (along with its Markov and confidence heads) is trained to match the backbone's predictions.
2. Post-training co-adaptation: During SFT, RL, and on-policy distillation, DSpark continues training alongside the evolving backbone policy so its draft proposals stay aligned with the post-trained model's reasoning and tool-use distributions, while stopping DSpark's gradients from flowing backward into the backbone weights.

keeping DSpark out of main pre-training simplifies the 45T-token pipeline while still providing fast speculative rollouts during RL/OPD training and production serving. (In the released config.json and inference/model.py, the DSpark parameters reuse legacy num_nextn_predict_layers config keys, and the minimal generate.py script runs standard single-token autoregressive decoding rather than the full speculative verification loop; see Chapter 26.)

Sources: Report §2.4.3 (pp. 13–14) and released inference/model.py (DSparkAttention, Markov/confidence heads, forward_spec) and inference/README.md. Chapter 16 turns from model architecture to training optimization across Muon, AdamW, and Sinkhorn-balanced updates.

16. Optimizing Different Parameter Families

A 748B-parameter model (552B backbone + 196B Engram) contains parameters with fundamentally different algebraic roles and memory footprints: 2D linear transformation matrices, multi-subspace attention projections, massive 196B lookup tables, and 1D normalization vectors. Instead of applying AdamW uniformly across the network, DeepSeek-V4.1-Flash partitions its parameters into four families optimized by Muon, head-wise Muon, momentum with Sinkhorn balancing, and AdamW.

Matching the optimizer to parameter geometry

Standard AdamW stores two full-precision moving-average buffers (first moment $m_t$ and second moment $v_t$) per parameter and scales each scalar coordinate independently. For 2D dense and expert matrices, independent element-wise scaling ignores the spectral structure of the linear map; for 196B-parameter Engram tables, maintaining two FP32 state buffers would consume over $1.5\text{ TB}$ of memory.

V4.1-Flash assigns each parameter tensor to one of four update rules:

1. Standard Muon: Applied to most 2D linear weight matrices (including MoE experts, output projections, and the linear patch projection in DeepSeek-ViT). Muon maintains a single momentum buffer ($\beta = 0.95$ with Nesterov lookahead), applies Newton–Schulz-style orthogonalization to precondition the 2D update matrix's singular values, rescales the orthogonalized update RMS by $0.18$ (to align with the AdamW learning-rate schedule), and applies decoupled weight decay of $0.1$.
2. Head-wise Muon: Applied to multi-head Query ($Q$) and Key ($K$) projection matrices. When all attention heads are stacked into a single concatenated matrix $W_Q$, running a single global orthogonalization across the stacked matrix couples the singular spectra of distinct attention heads. Head-wise Muon partitions the $Q$ and $K$ weight matrices along the head dimension and orthogonalizes each head's projection matrix independently, yielding consistent perplexity and downstream gains over monolithic Muon.
3. Momentum with Sinkhorn balancing: Applied to the token embedding table, the final language-model output head, and the 196B Engram tables (with no weight decay, and a $5\times$ learning-rate multiplier on Engram tables).
4. AdamW: Retained for 1D normalization weights, biases, gating scalars, and other non-matrix parameters ($\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 10^{-20}$, with weight decay $0.1$ on normalization weights and $0$ on biases/scales).

Sinkhorn-balanced updates for embeddings and Engram tables

In an embedding or Engram table of shape $m \times n$ (where each of the $m$ rows represents a token or n-gram entry and each of the $n$ columns represents a feature channel), standard Newton–Schulz orthogonalization is impractical because $m \gg n$ (for example, $m \approx 16\times 10^6$ rows and $n = 256$ channels per Engram table) and row gradients are sparse and heavily skewed by token frequency.

To precondition these tables using only a single momentum buffer $M_t$ (cutting optimizer state memory in half compared to AdamW's two buffers—saving ~784 GB of FP32 state across the 196B Engram parameters alone), V4.1-Flash uses Algorithm 1: Momentum with Sinkhorn Balancing:

1. Nesterov momentum accumulation: Given gradient $G_t$ at step $t$ and momentum buffer $M_{t-1}$, update $$M_t=\beta M_{t-1}+(1-\beta)G_t, \qquad \widehat G_t=\beta M_t+(1-\beta)G_t.$$
2. Alternating row/column $\ell_2$ normalization ($K = 11$ steps): Starting from $U^{(0)} = \widehat{G}_t$, alternate between normalizing the $\ell_2$ norm of each row and normalizing the $\ell_2$ norm of each column for an odd number of iterations ($K = 11$, so that both the first and final iterations normalize rows). To prevent near-zero gradient rows (from unvisited or rare n-grams) from having tiny numerical noise amplified into a unit-norm update, any row whose $\ell_2$ norm falls at or below $\tau = 10^{-3}$ times the mean row norm is masked to zero (with numerical stability floor $\epsilon = 10^{-20}$).
3. Width-aware scaling and step update: After the final row normalization, every active row of $U^{(K)}$ has unit Euclidean ($\ell_2$) norm, meaning its root-mean-square (RMS) across the $n$ feature columns is $1/\sqrt{n}$. Multiplying $U^{(K)}$ by $\sqrt{n}$ restores unit RMS per coordinate, and multiplying by $\gamma = 0.18$ matches the global AdamW/Muon step scale: $$W_{t+1} = W_t - \gamma \eta_t \sqrt{n}\, U^{(K)}.$$

Why alternate row and column normalization? Normalizing rows equalizes the update magnitude across frequent and rare token/n-gram rows, while normalizing columns prevents a handful of dominant feature channels from monopolizing the embedding update. Note that unlike the Sinkhorn–Knopp iterations in Single-Pass mHC (Chapter 13)—which normalize positive entries to sum to 1 ($\ell_1$) to form a doubly stochastic activation-mixing matrix—the optimizer's Sinkhorn procedure normalizes signed update matrices in Euclidean ($\ell_2$) norm to equalize row and column RMS.

In the distributed Engram implementation (§3.1.3), the 11 alternating normalization steps do not rewrite the full $m \times n$ matrix in HBM 11 times. Instead, the kernel maintains compact row-scaling and column-scaling vectors in memory and fuses the row-normalization pass with partial column-norm reduction.

Sources: Report §2.5 and Algorithm 1 (pp. 14–16), §3.1.3 (p. 18), and §4.2.2 (p. 22). Chapter 17 traces how these optimizers train the model across the 45T-token multimodal pre-training schedule.

17. Data Construction and the Pre-Training Run

DeepSeek-V4.1-Flash is pre-trained from scratch on 45 trillion tokens of interleaved text and multimodal data. Rather than treating corpus scaling as a raw volume target, the report focuses on maximizing marginal information gain per token, eliminating implicit synthetic duplication, and integrating native visual data at a 7:1 text-to-multimodal token ratio.

Text curation and contemporary code coverage

Adding high-scoring documents that repeat concepts already saturated in the corpus yields diminishing returns, whereas adding rarer technical or multilingual domains shifts capability scaling curves upward. To optimize data mixtures before committing to the full 45T-token run, the team used a multi-scale proxy ladder spanning model sizes and token budgets, evaluated across fine-grained domain benchmarks and expert-defined quality axes.

Two specific text curation shifts stand out in the report:

• Pruning low-information synthetic and translated web text: Modern web crawls contain growing volumes of SEO-farmed LLM outputs and low-quality machine translations that act as semantic duplicates of existing high-quality documents. The pipeline explicitly detects and strips these low-information synthetic pages (while retaining targeted model-in-the-loop data curation and downstream agent task synthesis).
• Expanding contemporary software repositories: The code corpus incorporates fresher repositories, commit histories, modern library ecosystems, and underrepresented programming languages, providing stronger priors for downstream repository-level coding agents.

Multimodal data pipeline and staged filtering

Multimodal pre-training data draws from three primary sources:

1. Image-text pairs: Extracted from web images and alt-text, filtered for semantic alignment and deduplicated in visual embedding space.
2. Interleaved image-text documents: Parsed from web pages and PDFs with their natural reading order and layout preserved. Because the team's legacy web crawler had been tuned for text density and systematically under-sampled image-rich pages, they re-bootstrapped their multimodal crawl directly from raw Common Crawl snapshots.
3. Domain-specific visual datasets: Targeted collections covering visual grounding, UI element pointing, dense document OCR, charts, image-to-code pairs (such as HTML/CSS/SVG rendering), and multi-step computer-use trajectories.

Because downloading, decoding, and running vision models on billions of raw web images is prohibitively expensive in CPU and I/O bandwidth, the multimodal pipeline uses a staged filtering funnel: cheap text heuristics, statistical filters, and URL/text deduplication run before fetching image bytes. Surviving documents then undergo image-aware filtering and layout assembly, followed by fine-grained multimodal quality scoring using SmolVLM. Interleaved documents that fail full-document coherence checks but contain valuable individual figures are salvaged by extracting and re-pairing their high-scoring image-text segments.

Corpus unification and low-waste sequence packing

The text-only and multimodal corpora are curated in separate pipelines and merged into a unified training corpus at a 7:1 ratio of text-only tokens to multimodal tokens (at 45T total tokens, a uniform 7:1 split corresponds to roughly $39.375\text{T}$ text-only tokens and $5.625\text{T}$ multimodal tokens). Whenever the same underlying web document appears in both the text-only and multimodal pools, the pipeline drops the text-only version in favor of the interleaved multimodal version and assigns it the higher of the two configured epoch weights.

To pack variable-length documents into 64K and 1M training sequences without wasting compute on padding tokens or causing data imbalance across distributed ranks:

• Ultra-long documents are deterministically pre-split prior to shard mixing so that token length distributions remain uniform across training steps and context-parallel ranks.
• Sample assignment is coordinated jointly across the main pre-training phase and the long-context extension phase so long documents are not inadvertently duplicated across phases.
• An enhanced best-fit sequence packing algorithm packs documents with sample-level causal attention masks (preventing cross-document contamination within a packed sequence) while holding the wasted padding token rate to $\le 10^{-4}$ ($0.01\%$).

The 45T-token pre-training schedule and 1M context extension

Main pre-training runs with a constant global batch size of 100.6M tokens across four learning-rate phases (roughly $447{,}316$ optimization steps over 45T tokens):

Two architectural milestones occur at specific points along this 45T-token timeline:

1. Vision encoder unfreezing at 28T tokens: As described in Chapter 5, DeepSeek-ViT (pre-trained on 47B contrastive pairs and 236B generative alignment tokens with a 4B proxy MoE) starts the main run frozen except for its final norm and MLP projector. When learning-rate decay begins at 28T tokens, the entire DeepSeek-ViT encoder is unfrozen and trained jointly with the backbone at a reduced vision learning rate.
2. Sparse-from-scratch training and 1M context extension at 34T tokens: Unlike models that pre-train with dense attention and convert to sparse attention late in training, V4.1-Flash trains with CSA2 sparse attention from step 0 at a 64K context length. At 34T tokens (inside the cosine decay phase), the training sequence length is extended from 64K to 1M tokens for the remaining 11T tokens ($24.4\%$ of the total pre-training budget). The report notes that the entire 45T-token run completed without loss spikes or training instabilities.

Following the 45T-token run, the frozen backbone is used to train the 3-block DSpark drafter (Chapter 15), and the post-training pipeline (Chapter 20) introduces hierarchical sparse indexing, FP4 main-cache QAT, and simulated decoder bounded replay.

Sources: Report §§4.1–4.2 (pp. 20–23), §2.3.2 (p. 11), and §§2.4.3–2.4.4 (pp. 13–14). Chapter 18 examines the distributed systems engineering required to train this multimodal, cross-layer-shared architecture at scale.

18. Distributed Training Infrastructure

Training DeepSeek-V4.1-Flash across thousands of accelerators introduces three systems challenges absent from standard text-only Transformers: variable-resolution visual inputs that create load imbalance and host I/O bottlenecks, CSA2 KV banks and index selections shared across pipeline-parallel stage boundaries, and 196B-parameter Engram lookup tables that must be prefetched and updated without stalling the pipeline.

Overlapping communication in contrastive vision pre-training

During Stage 1 preparation of DeepSeek-ViT (47B image-text pairs under the SigLIP contrastive objective), every image embedding must be compared against every text embedding in the global batch, requiring all-gather collectives across data-parallel (DP) ranks. In the backward pass, the gradient with respect to local text features depends on the gathered global visual features, while the gradient with respect to local visual features depends on the gathered global text features.

Rather than blocking on synchronous all-gathers before computing losses, the training runtime pipelines the two modalities:

• It launches the all-gather of visual features asynchronously during the text encoder's forward pass.
• It launches the all-gather of text features asynchronously during the text encoder's backward pass.

By hiding both global feature collectives behind text-encoder compute, the SigLIP stage eliminates cross-rank communication stalls.

Disaggregated vision execution and context-parallel image sharding

During the main 45T-token multimodal pre-training run, interleaved sequences contain highly variable numbers of images per sample. Embedding DeepSeek-ViT directly inside the first pipeline stage of the language backbone would cause severe pipeline bubbles whenever a microbatch hits an image-heavy sequence.

Instead, the infrastructure disaggregates DeepSeek-ViT from the language pipeline, replicating the vision encoder outside the LLM's pipeline-parallel parameter tree and executing each training step in three coordinated phases:

1. Phase 1 (Vision Forward): Balanced batches of images run through DeepSeek-ViT across the cluster to produce visual embeddings.
2. Phase 2 (LLM Forward & Backward): The 40-layer language backbone runs its standard pipeline/expert/context-parallel schedule over the combined text and visual embeddings, returning gradients with respect to the visual embeddings.
3. Phase 3 (Vision Backward): When DeepSeek-ViT is unfrozen (from 28T to 45T tokens), the accumulated visual embedding gradients propagate backward through DeepSeek-ViT.

At 1M-token context lengths, a single packed sequence can also contain hundreds of high-resolution images—enough raw image bytes to overwhelm a single host node's CPU decoding threads, DRAM, and storage I/O if loaded on one machine. The dataloader therefore shards images across context-parallel (CP) ranks, ensuring each image in a 1M-token sequence is fetched and decoded from storage by exactly one host and balanced across the CP group.

To see when host image I/O remains completely hidden behind GPU compute, let $\rho$ denote the average raw image bytes per token, $B_{\mathrm{IO}}$ the per-node storage ingestion bandwidth (bytes/s), $C$ the GPU compute work per token, and $R_{\mathrm{GPU}}$ the GPU compute throughput. For a sequence of $N$ tokens, host loading time $N\rho / B_{\mathrm{IO}}$ is smaller than GPU compute time $NC / R_{\mathrm{GPU}}$ whenever

Because $N$ cancels on both sides once images are evenly sharded across CP ranks, a full-scale 552B MoE training run (large $C$) stays comfortably compute-bound even on image-dense sequences, whereas tiny proxy ablation models (small $C$) can become storage-I/O-bound unless preprocessed features are cached. During multi-turn RL rollouts, the engine similarly transmits only newly added images at each turn and caches decoded/preprocessed visual tensors in distributed storage so rollout workers and training workers never decode the same image twice.

Cross-stage CSA2 sharing and shadow indexers

In a pipeline-parallel (PP) deployment of the 40-layer backbone, the 6-layer encoder groups and the 20-layer shared decoder KV group inevitably span multiple pipeline stages:

• A Full layer on Stage $A$ creates a global KV bank and indexer keys that must be read by Reuse and Reindex layers on downstream Stage $B$ (and $C$).
• During the backward pass, gradients from all consumer layers on downstream stages must accumulate back into the source projections on Stage $A$.

To handle this without extra cross-stage round-trip messages for index lookups, the training system passes the compressed KV bank and routing metadata forward as part of the standard point-to-point pipeline activation payload (partitioned by context parallelism so full 1M-token banks are never replicated onto a single GPU). When a group spans pipeline stages, the runtime instantiates shadow indexers—executable replicas of the indexer module on the consumer stages that share a single logical parameter owner. The owning stage manages the master weights, optimizer states, and checkpoint serialization, while gradient reductions and parameter broadcasts keep the shadow replicas synchronized at every step.

Because multiple microbatches are simultaneously in flight inside a 1F1B or interleaved pipeline schedule, a dedicated microbatch state manager tracks the lifecycle of each microbatch's shared KV banks and index lists across forward execution, activation checkpoint recomputation, and backward propagation, freeing the buffers only after the final consumer layer on that stage completes its backward pass.

Sharded Engram training

As introduced in Chapter 14 and Chapter 16, the two Engram modules at layers 1 and 14 hold 196B parameters across 48 hash tables. During training, each table is row-sharded across dedicated Engram parallel groups, with the single momentum buffer of the Sinkhorn-balanced optimizer further sharded across data-parallel replicas.

Because n-gram hash addresses depend only on input token IDs, each pipeline stage prefetches the FP8 embedding rows and scales for the entire local batch before microbatch execution starts, feeding the prefetched FP8 tensors directly into the gate/value projection GEMMs. During the backward pass, sparse table gradients are buffered locally as microbatches finish and flushed back to the owning Engram ranks after the LLM backward pass completes, overlapping the Engram gradient all-to-all transfer with Phase 1 and Phase 3 of DeepSeek-ViT execution.

Sources: Report §3.1 (pp. 16–18). Chapter 19 shifts from pre-training systems to how V4.1-Flash synthesizes and verifies interactive agent environments for post-training.

19. Making Agent Tasks and Environments

In §5 of the report, the authors state directly that DeepSeek-V4.1-Flash introduces no new post-training optimization algorithm; instead, nearly all of its post-training gains come from scaling the quality, diversity, and verifiability of interactive agent environments. Rather than training solely on static prompt-response pairs, the post-training pipeline constructs executable task triplets at million-environment scale.

The (Problem, Environment, Verifier) triplet

Every agent training task is formalized as a coupled triplet:

A flaw in any one of the three components corrupts the reinforcement learning signal:

• If the environment has missing dependencies, broken build scripts, or leaked git commits containing the ground-truth patch, the agent either fails on setup noise or learns to cheat by scraping git history.
• If the problem specification omits implicit requirements that the test suite checks (for example, asking to "add CSV export" while the test harness asserts a specific unmentioned filename and column header format), RL rewards guessing hidden test implementation details rather than following instructions.
• If the verification system has incomplete test coverage, a policy can pass the reward check with a degenerate stub or by breaking existing functionality.

Task synthesis therefore optimizes two criteria jointly: calibrated difficulty (nontrivial for the current policy, so rollouts generate informative contrast between passing and failing trajectories) and strict triplet consistency (verified by independent multi-agent auditing).

Reconstructing general-agent and enterprise workflows

To train general tool-use and enterprise workflows beyond standard coding benchmarks, DeepSeek collected voluntary interaction logs and failure reports from internal employees and external enterprise partners. From these real-world transcripts, an automated synthesis pipeline reconstructs executable mock environments that replicate the exact tool schemas, input/output JSON contracts, stateful database behaviors, and error modes of common SaaS platforms and internal business backends.

Focusing environment synthesis on interaction turns where previous models failed or received negative user feedback ensures that the mocked sandboxes target genuine multi-turn failure modes—such as handling partial API failures, schema mismatches, and ambiguous user constraints—in a deterministic, replayable container.

Automated multi-agent pipeline for coding environments

For software engineering tasks, the pipeline mines both difficult real-world coding sessions (deduplicated at the trajectory level) and public GitHub repositories above a star-count threshold, processing each candidate through a five-stage multi-agent factory:

1. Feasibility & specification agent: Inspects the repository or session history to verify that the project can be built and tested inside an isolated Linux container, selects a starting commit or turn, synthesizes a complex feature or bug-fix specification, and defines both fail-to-pass (F2P) tests (which fail on the base commit and pass when the feature/fix is implemented) and pass-to-pass (P2P) regression tests.
2. Environment builder & sanitizer agent: Provisions dependencies, compiles the workspace, runs self-checks on the F2P/P2P test suites, strips all solution-revealing traces (such as future git commits, reflogs, cached build artifacts, or issue comments containing patches), and snapshots the clean workspace into a container image layer.
3. Solver ensemble: Multiple distinct agent policies attempt to solve the packaged task inside the sandbox, generating a diverse set of candidate trajectories.
4. Independent inspector agent: Audits the problem description, container state, verifier scripts, and solver trajectories together—flagging mismatches between instructions and assertions, flaky or unbuildable dependencies, and reward-hacking shortcuts.
5. Repair & calibration loop: Tasks flagged by the inspector are routed to a repair agent that fixes broken tests, tightens underspecified instructions, or recalibrates difficulty before sending the task back for re-verification.

Crucially, task auditing does not end once an environment enters the training pool. Every time the RL policy improves and starts a new training iteration, fresh solver rollouts are monitored for anomalous pass rates or suspiciously short trajectories, catching newly discovered reward hacks (such as exploiting local package mirrors or test-runner internals; see Chapter 21).

Sources: Report §5.1 and §5.1.1 (pp. 25–26). Chapter 20 explains how trajectories sampled from these verified environments are used across SFT, RL, specialist teacher training, and on-policy distillation.

20. SFT, Reinforcement Learning, and On-Policy Distillation

While the 45T-token pre-training run builds broad linguistic, visual, and coding knowledge, turning the base model into an interactive agent requires teaching it to follow multi-turn tool protocols, recover from execution errors, calibrate its reasoning length, and stop cleanly once a task is verified. DeepSeek-V4.1-Flash accomplishes this through a three-stage post-training pipeline—Supervised Fine-Tuning (SFT), large-scale asynchronous Reinforcement Learning (RL), and full-vocabulary On-Policy Distillation (OPD)—while simultaneously adapting the backbone to its production inference approximations.