Introduction

DeepSeek-V4 is a family of language models built to make very long contexts affordable. The paper combines changes to attention, residual connections, optimization, training, and serving. Understanding any one of these changes helps. Understanding how they fit together explains the model.

This book reads the paper as an engineering and research document. It explains the mechanisms, works through their costs, and asks what the results establish. It also identifies details the report leaves open.

The main source is the TeX snapshot in this directory's parent, titled DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. The snapshot describes preview models. Its benchmark results, hardware observations, and model descriptions are treated as claims about that snapshot. They are not a live ranking or a guarantee about a later API.

How to read the book

Read Chapters 1–3 for the problem, vocabulary, and overall design. Chapters 4–10 explain the model's main mechanisms. Chapters 11–17 explain how the model is trained and operated. Chapters 18–20 examine the evidence and what it means for use. Chapters 21–22 provide a worked trace and a reference.

Each chapter starts with the concepts it needs. Some terms recur because a reader should be able to open an attention chapter or a training chapter directly. Equations come with explanations and, where useful, small examples.

Three kinds of statements appear throughout:

• Reported: a fact or measurement stated in the local paper or its tables.
• Derived: arithmetic or a mathematical consequence of a stated design, with assumptions given.
• Interpretation: an explanation of why a design may help, or what evidence would be needed to assess it.

Unmarked explanations of standard concepts are background. An explanation of a possible mechanism should not be mistaken for an ablation result.

Contents

Reading paths

For architecture, read 2–10, then 21. For training, read 3, 9–12, and 14–16. For inference engineering, read 5–8, 12–13, and 16–17. For evaluation or model selection, read 1, 3, and 18–20.

These paths are shortcuts through the material, not separate summaries. The later chapters still refer back to the design decisions that explain the measurements.

Scope and source discipline

This book does not claim to reproduce the training run. The paper omits parts of the recipe, including exact data proportions, several training budgets, and some details inherited from earlier DeepSeek reports. It also does not provide a complete set of controlled ablations for the combined design.

Numerical examples use decimal units unless stated otherwise: 1M means 1,000,000; one GB means $10^9$ bytes. The released Pro configuration uses a maximum position count of 1,048,576, so a calculation at exactly one million tokens is an approximation to that limit. Chapter 22 records this small distinction.

Internal chapter links use the documentation site's routes. Source references name the local TeX file, section, table, and line range so they remain auditable against the supplied files. Chapter 22 records the source checksum and the limited implementation checks used to supplement the paper.

1. Why a Million Tokens Changes the Problem

A language model reads a sequence of tokens and predicts what comes next. As the sequence grows, the model must spend more effort reading its past. DeepSeek-V4 starts from a practical problem: a useful agent or document-analysis session can become much longer than a conventional chat.

The paper's answer is to change how the model represents and reads earlier tokens. It compresses old information, selects some of that compressed information, and keeps recent detail directly accessible.

What a million-token context permits

A context window is the maximum sequence the model can condition on in one run. It contains more than the current question. Depending on the application, it can include system instructions, previous messages, retrieved documents, tool results, and the model's generated tokens.

Consider an agent modifying a large codebase. It reads files, runs tests, receives long logs, revises a patch, and returns to an earlier decision. Every round adds material. A long window lets the application preserve more of that history without repeatedly summarizing or discarding it.

For document analysis, a large window can hold many source documents at once. That can help answer questions whose evidence spans several reports. It can also preserve the original wording so a later question can ask about a detail that an earlier summary omitted.

Neither use case makes context length equivalent to intelligence. The model still has to identify relevant material, resolve conflicting sources, and reason correctly. Adding documents can even make these tasks harder by increasing distraction.

The distinction is between capacity to include information and ability to use it. A one-million-token window establishes the first. Retrieval and question-answering evaluations test parts of the second.

The two phases of inference

Inference usually has two phases.

Prefill processes the prompt's tokens. The model computes hidden states and prepares cached information for each layer. A long prompt can make this phase expensive before the first answer token appears.

Decode generates new tokens one at a time. Each new token uses the cached past and adds new state to it.

The user notices both costs. Time to first token depends heavily on prefill, scheduling, and cache reuse. The speed of the rest of the answer depends on decode latency. A model can improve one phase more than the other.

The V4 paper's headline FLOP comparison concerns single-token inference at a given context length. It should not be read as a measurement of the time to prefill that entire context, or of the total wall-clock time for a task.

Why ordinary attention becomes expensive

Attention lets a token consult other tokens. In a conventional causal decoder, token $t$ can read positions up to $t$. Processing all tokens produces roughly

query–key interactions. The attention part of prefill therefore grows quadratically with sequence length.

For decode, the new token reads the existing past once. Its attention work grows roughly linearly with $N$. Generating $T$ new tokens after a prompt of length $N$ involves approximately

past-reading interactions per head, before accounting for dimensions and layers.

This matters for reasoning. A model that produces a long solution does more than pay for extra output tokens. Later reasoning tokens also read a longer past. A cheaper way to read history makes a larger reasoning budget more feasible.

There is a separate memory problem. A server keeps key–value caches, or KV caches, so it does not have to recompute the entire past at each decode step. These caches grow linearly with context length. A server hosting many long requests can run out of cache memory even when its weights fit comfortably.

Quadratic attention computation and linear cache growth are different problems. An attention kernel can reduce temporary memory without eliminating the number of interactions. A sparse selector can reduce the interactions without eliminating the stored past. Sequence compression can reduce the stored past itself.

V4 combines these approaches because none alone addresses every cost.

Three ways to make history cheaper

The first approach is sharing. Multiple query heads can read the same keys and values. This reduces stored KV dimensions. V4 uses a shared KV representation.

The second is selection. A query can read a subset of history rather than every entry. V4's Compressed Sparse Attention, CSA, selects high-scoring entries.

The third is compression. Several tokens can be represented by one learned vector. Both CSA and Heavily Compressed Attention, HCA, compress along the sequence dimension.

These operations answer different questions:

V4 also keeps a small uncompressed sliding window. This preserves recent information and allows access to tokens in an incomplete compression block.

Why there are two compressed attention types

CSA compresses at a modest rate and then selects entries. In the reported configuration, four original tokens produce one new compressed entry. The compressor overlaps adjacent blocks, so the entry can draw from eight source tokens while the sequence still shrinks by a factor of four.

HCA compresses much more heavily: 128 original tokens produce one entry. A query reads all eligible HCA entries, so HCA is dense over a much shorter sequence.

These choices preserve different forms of access. CSA has more detailed memory but reads it selectively. HCA has coarser memory but reads broadly. The models interleave them across layers.

This is a plausible division of work, not proof that each type learns a clean, separate role. The paper describes the architecture and reports the full model's results. It does not establish a complete causal account of what each layer represents.

What the headline efficiency claims mean

At a one-million-token context, the paper reports the following comparisons with DeepSeek-V3.2:

The FLOPs are expressed as equivalent FP8 FLOPs. That convention accounts for precision choices in the cost estimate. It is not a direct measurement of token latency.

The comparison is striking because Pro activates more parameters than V3.2: 49B versus 37B. This illustrates how, at very long contexts, attention can outweigh the benefit of a smaller feed-forward computation. A model can use more active weights and still perform less total work per generated token if its history-reading cost is much lower.

The ratios are also context-dependent. At short context lengths, attention has fewer past entries to read. Weight access, projection matrices, feed-forward networks, and overhead become a larger fraction of total cost. A million-token ratio should not be applied to a thousand-token chat.

The question of information loss

Compression is not a free substitute for the original text. A single vector generally cannot preserve every independent detail from all the tokens it summarizes. The useful question is whether training teaches the compressor to preserve enough information for later tasks.

For a legal or scientific passage, the important information may include a negation, a date, a numerical value, or the relation between two entities. A compressor that retains the topic but loses the negation can cause a serious error. A selector that misses the relevant block can cause the same failure even if the compressor preserved it.

Long-context evaluation therefore needs several tasks. Synthetic retrieval can test whether specific items survive and remain findable. Cross-document questions can test whether evidence is combined. Real workloads can reveal how distractors, repeated text, and changing instructions affect behavior.

V4's long-context results are evidence that the design is useful. They do not show lossless recall of arbitrary text.

What the rest of the paper must solve

An efficient attention formula still needs a trainable model and a workable system. V4 adds mHC to the residual stream, Muon to optimization, stability controls for MoE training, and specialized kernels and caches.

Post-training adds another layer. Domain specialists learn with supervised fine-tuning and reinforcement learning. A unified model then learns from those specialists through on-policy distillation. Low-precision training prepares the model for the representation used in serving.

The model's final behavior is the result of this whole process. To understand the paper, we need to follow the path from a token's representation to its memory, from gradients to stable training, and from benchmark scores to practical limits.

Source: local main.tex, Introduction, lines 190–270; attention overview and efficiency discussion, lines 365–385 and 533–545. The numerical comparisons are reported estimates, not measurements made for this book.

Continue with the foundations or return to the contents.

2. The Concepts the Paper Assumes

DeepSeek-V4 is a decoder language model. It predicts tokens using representations built by a stack of Transformer blocks. This chapter defines the terms used in later chapters and separates several ideas that are easy to confuse.

Tokens and hidden states

A tokenizer maps text to a sequence of integer IDs. Tokens may represent words, parts of words, punctuation, or other byte sequences. A million tokens is not a million words. The relation depends on language, formatting, and the tokenizer.

An embedding table maps each token ID to a vector. This vector is the starting representation of the token. The model repeatedly transforms it as information flows through the layers.

For a sequence of $N$ tokens and hidden width $d$, a layer's input is commonly written

Row $t$ is the representation of token $t$. The width $d$ is the number of coordinates available to represent it. A coordinate is not assigned a fixed human-readable meaning.

The output head maps the final hidden state to one logit per vocabulary item. Softmax converts these logits into probabilities:

The model samples or selects a next token from this distribution. A new forward pass then uses the enlarged sequence.

What an attention head computes

An attention head produces a query from the current token and compares it with keys associated with accessible past positions. It uses the resulting weights to combine values from those positions.

A simplified formula is

where $c$ is the head dimension. The mask prevents reading disallowed positions, including future tokens in a causal model.

A query asks, in a learned numerical form, what information the current computation needs. A key describes how an entry can be matched. A value carries information read from that entry. These are training-based roles, not explicit labels written by a programmer.

Many heads allow several forms of matching at once. Their outputs are combined and projected back to the model's hidden width.

V4 modifies the objects being matched. Some keys and values represent compressed groups of tokens rather than individual positions. It also uses one shared vector as both key and value for an entry.

MHA, GQA, MQA, and MLA

These names mostly describe how attention heads share or compress representations.

Multi-head attention, MHA, gives each query head its own key and value head. This provides independent representations but can produce a large cache.

Grouped-query attention, GQA, groups query heads so each group shares a KV head. If 64 query heads share eight KV heads, the cache stores eight KV representations per token.

Multi-query attention, MQA, gives all query heads one shared KV head. Query heads remain different even though they read a common memory.

Multi-head latent attention, MLA, used in earlier DeepSeek models, represents KV information in a lower-dimensional latent form and uses the structure of the projections to reduce cache storage.

These descriptions do not fully define an implementation. Positional dimensions, projection shapes, and cached formats matter. V4 retains low-rank query generation but describes its core attention as shared key–value MQA. Calling it ordinary MLA obscures the change.

Sequence compression is another axis. MQA reduces how many KV heads are stored. V4's compressors reduce how many sequence entries are stored. The two savings can be combined.

Why the KV cache exists

At decode time, previous tokens have already been processed. Their keys and values can be retained. A new token computes its query and its new KV information, reads the cache, and extends it.

For an ordinary implementation with $L$ layers, $H_{\mathrm{KV}}$ KV heads, head dimension $c$, and $b$ bytes per scalar, the payload is roughly

The factor two is for separate keys and values. V4's tied KV representation removes that factor in its main stored entry. Compression further changes the number of entries.

This is only payload memory. Scale factors, allocation granularity, metadata, temporary workspace, and partially completed requests add overhead.

The cache is not the original prompt stored in a readable form. It is an internal representation produced separately at each layer. A server can keep the text transcript alongside it, but the two are different objects.

Residual connections

A residual connection adds a layer's output to its input:

This gives information and gradients a path across many layers. Each block can modify a representation without having to reconstruct it completely.

V4's mHC expands the residual stream into several vectors per token. It learns how to combine them before a sublayer, how to mix them along the residual path, and where to place the sublayer's output. The expensive attention and feed-forward computations still receive the usual hidden width.

Thus, residual width and hidden width become separate choices. A four-stream residual does not mean four complete attention modules.

Normalization and position

RMSNorm rescales a vector using its root mean square:

Here $g$ is a learned scale and $\epsilon$ prevents division by zero. Unlike mean-centering normalization, this expression does not subtract the vector's mean.

V4 normalizes queries and compressed KV entries before attention. This helps control the size of attention logits. Normalization does not make all logits equal, and learned scales still influence their magnitude.

Rotary positional embeddings, RoPE, rotate pairs of coordinates according to position. The dot product of rotated queries and keys depends on relative position. V4 applies RoPE only to part of each attention vector and also reverses the query's rotation on the attention output. Chapter 7 explains why tying keys and values makes the latter operation useful.

Dense computation and sparse computation

The word dense has several meanings in this paper.

A dense feed-forward network uses the same weights for every token. An MoE network routes each token to a subset of expert networks.

Dense attention reads every eligible memory entry. Sparse attention reads a subset.

A dense matrix operation multiplies ordinary matrices without exploiting many zero entries.

HCA is dense attention over a compressed sequence. The overall model is still an MoE model. Calling HCA dense does not mean it stores every original token, or that the model is a dense-parameter model.

The word expert also has two meanings. An MoE expert is a feed-forward subnetwork inside a block. A post-training domain specialist is a separate trained model. More than ten domain teachers do not mean an MoE layer has ten extra feed-forward experts.

Training stages

Pre-training usually teaches broad language prediction from a large corpus. A typical objective maximizes the probability of the next observed token.

Supervised fine-tuning, SFT, trains on prompt–response examples chosen to teach useful behavior, formats, and skills.

Reinforcement learning, RL, generates responses and changes the policy using a reward signal. The reward can come from a test, a formal verifier, or a learned evaluator.

Distillation teaches a student to match a teacher's behavior or probability distributions. V4's final consolidation uses responses sampled from the student itself, then evaluated by relevant teachers. This is on-policy distillation.

Quantization-aware training, QAT, exposes the model to a low-precision representation during training so it can adapt before deployment.

The stages solve different problems. More pre-training data does not automatically establish reliable tool formatting. RL can improve a rewarded skill without automatically improving another skill. Distillation depends on teacher quality and on what the student is allowed to experience.

Policy, trajectory, and on-policy data

A policy is the model's distribution over possible actions. For ordinary generation, the action is a token. For an agent, the generated text can encode a tool call, making the token sequence part of a larger action sequence.

A trajectory is the sequence of generated decisions and resulting observations. In a coding task, it may include reasoning, shell commands, test results, and file edits.

On-policy data comes from the policy being trained, or a sufficiently close version of it under the method's assumptions. Off-policy data comes from another policy.

This distinction matters because a student can make choices its teacher would not make. Training only on teacher-written solutions leaves the student less prepared for states created by its own errors.

FLOPs, bytes, and time

FLOPs count floating-point operations. Memory bytes count stored or transferred data. Latency measures elapsed time. None is a universal substitute for the others.

A matrix multiplication may be limited by arithmetic throughput. A decode pass through a huge set of weights may be limited by memory bandwidth. A distributed MoE operation may be limited by network communication or synchronization.

A design that reduces FLOPs can still have irregular memory access. A design that reduces bytes can still need an expensive selector. A design that helps one request can affect batch utilization differently.

Later chapters keep these quantities separate. That separation is necessary to read V4's efficiency claims correctly.

Source: background concepts are explained here for the reader. V4-specific uses follow local main.tex, Architecture, lines 278–593; Pre-Training and Post-Training, lines 904–1294.

Continue with the model map or return to the contents.

3. The Model in One View

DeepSeek-V4 has two reported model sizes: Flash and Pro. Both are Mixture-of-Experts models with a million-token context. Both combine compressed attention, multiple residual streams, and a shared training approach.

The difference is more than total parameter count. Pro has more layers, wider hidden states, more query heads, larger experts, and a larger sparse selection budget.

The configuration table

The following values come from the paper's model setup. They are architectural settings, not suggested serving parameters.

Flash's first two layers use pure sliding-window attention. Pro's first two layers use HCA. Later layers interleave CSA and HCA.

The first two layers and the first three MoE layers describe different parts of each block. The attention choice and the routing choice should not be merged into one rule.

A block's two kinds of computation

A Transformer block generally contains attention and a feed-forward transformation. Attention allows interaction across sequence positions. The feed-forward transformation processes the representation at each position.

V4 uses an MoE feed-forward module in every Transformer block. A token passes through a shared expert and selected routed experts. In the first three MoE layers, token IDs determine routing through a predefined hash rule; later routing is learned from the token's representation.

The residual stream connects these transformations. With mHC, a token carries four residual vectors rather than one. Learned mappings produce the input to each expensive sublayer and combine its output with the continuing residual state.

A conceptual path is:

token ID
  → embedding
  → expanded residual state
  → repeated attention and MoE transformations with mHC connections
  → final representation
  → vocabulary logits
  → next-token distribution

This is a structural guide. It does not specify every norm placement, tensor layout, or final stream-reduction operation. The paper refers readers to the implementation for details it omits.

The three lengths inside attention

At a context length $N$, it is useful to distinguish three sequence lengths.

The original sequence has $N$ token positions. The CSA compressed sequence has roughly $N/4$ entries. The HCA compressed sequence has roughly $N/128$ entries.

A CSA query does not read all $N/4$ main entries. An indexer scores eligible compressed blocks and selects at most $k$ of them. The query then reads those selected entries along with up to 128 recent uncompressed entries.

An HCA query reads all eligible HCA entries along with the local window. At a million original tokens, that is roughly 7,812 compressed entries, plus the window.

Neither operation reads a million full-size main KV entries per query. The CSA indexer still scans the compressed candidate sequence, which is why a fixed main-attention budget does not make all attention work constant.

Where the query heads differ from the KV head

Flash has 64 main query heads; Pro has 128. Each head has a 512-dimensional query and output. The main KV representation is shared.

The concatenated attention outputs therefore have widths

for Flash and

for Pro.

These are much wider than the model's hidden states. V4 uses grouped output projections to reduce the cost of mapping those outputs back to $d$ dimensions. The group counts keep eight query heads per group in both sizes.

Query head count affects compute and output projection size. It does not multiply the stored main KV entry count in the way ordinary MHA would.

What 1.6T parameters means

The total parameter count measures the model's learned storage. Activated parameters measure the portion involved in a token's forward computation under the routing scheme.

For Pro, six of 384 routed experts are selected in each MoE layer. For Flash, six of 256 are selected. There is also a shared expert and common attention, routing, embedding, and output machinery.

It is therefore wrong to calculate activated parameters as total parameters multiplied by $6/384$ or $6/256$. Those fractions apply only to the routed-expert portion of an MoE layer.

The distinction explains two practical facts. First, the model can have a much larger knowledge-bearing parameter store than a dense model with similar per-token arithmetic. Second, the unselected weights still have to be stored somewhere. A 49B-active model can require a deployment sized for a far larger weight set.

Across a batch, different tokens select different experts. The set of weights touched by the batch can be much larger than the set touched by one token.

What stays from earlier DeepSeek models

The paper retains the DeepSeekMoE framework, auxiliary-loss-free load balancing with a small sequence-level balance loss, and multi-token prediction. It also inherits many unspecified details from DeepSeek-V3.

V4 changes the routing score transformation from sigmoid to the square root of softplus. It removes a constraint on the number of routing target nodes. It replaces initial dense feed-forward layers with hash-routed MoE layers.

Attention is a larger change. CSA combines sequence compression with DeepSeek Sparse Attention. HCA uses heavier compression with dense attention over the compressed sequence.

The optimizer also changes for most matrices: Muon replaces AdamW there. AdamW remains for specified modules and small parameters.

“Inherited” does not mean every inherited mechanism is fully redefined in this report. A reproduction effort would need the earlier reports and implementation as well as V4's source.

Three reasoning modes

Each model supports Non-think, Think High, and Think Max. These are behavior settings taught in post-training. The paper describes distinct length penalties and context windows for specialist RL, plus response formats and a special system instruction for Max.

A mode is not another parameter scale. V4-Pro-Max and V4-Pro-High refer to the Pro model under different reasoning effort settings.

The evaluation windows for knowledge and reasoning are 8K, 128K, and 384K for Non-think, High, and Max respectively. Agent evaluation uses a separate maximum context setting of 512K. The architectural context capacity remains roughly one million tokens.

This distinction prevents a common mistake: assuming every benchmark uses the maximum supported context, or that a larger context budget alone explains a reasoning score.

The lifecycle of the model

The reported pipeline has several stages:

1. Construct and process the pre-training corpus.
2. Train the base model, gradually extending context length and introducing attention sparsity.
3. Train domain specialists with SFT and RL.
4. Consolidate specialist behavior into one student using multi-teacher on-policy distillation.
5. Use quantization-aware post-training to prepare expert weights and the CSA indexer for FP4 deployment.

The last two items are related parts of post-training. This list does not imply the paper specifies one strictly ordered, fully disclosed schedule for every specialist, teacher, and quantization change.

Serving then uses specialized kernels, heterogeneous KV caches, and prefix reuse. The implementation determines how much of the architecture's theoretical saving appears as user-visible speed.

What this map does not settle

The configuration table cannot tell us how much each component contributes to quality. A larger model also has different training scale and settings. A comparison between Flash and Pro is not a clean ablation of layer count, head count, or expert count.

The table also cannot determine a complete deployment footprint. Actual precision, scales, padding, sharding, cache allocation, and runtime workspace must be known.

The map is most useful as a set of constraints: any explanation of the model must fit these dimensions, and any efficiency claim must account for the operations they imply.

Source: local main.tex, Architecture, lines 278–305; Model Setups, lines 923–957; reasoning modes and evaluation settings, lines 1063–1110 and 1363–1388. Chapter 22 notes small differences between the paper's rounded descriptions and released configuration values.

Continue with experts and multi-token prediction or return to the contents.

4. Experts and Multi-Token Prediction

An MoE model keeps many feed-forward networks but uses only a few for each token. This lets it increase total capacity without increasing per-token computation by the same amount. DeepSeek-V4 retains this design and changes several routing details.

This chapter concerns experts inside the model. The domain specialists used in post-training are separate models and are discussed in Chapter 14.

What an expert computes

A Transformer feed-forward module transforms a token's hidden vector independently of other sequence positions. A SwiGLU expert uses two input projections and one output projection. A simplified expression is

If the hidden width is $d$ and the intermediate width is $f$, the three matrices contain roughly $3df$ weights. Ignoring activation and overhead, a token–expert pair uses roughly $6df$ FLOPs under the convention that a multiply and an add count as two.

For Pro, $d=7168$ and $f=3072$. One such expert therefore contains about

weights. This is a derived count for the three principal matrices, not a complete checkpoint inventory.

An MoE layer replaces one feed-forward network with several of these experts and a routing mechanism. In V4, one shared expert is used for every token, while six routed experts are selected for each token.

Shared and routed experts

A shared expert provides a common transformation. Routed experts provide conditional transformations. In a schematic formulation,

where $S(h)$ is the selected expert set and $a_e(h)$ is a routing weight.

This expression explains the design but omits exact scaling and normalization details. The paper states that unspecified MoE settings follow the earlier DeepSeek design.

Shared computation can reduce the need for routed experts to duplicate every common operation. Fine-grained routed experts can provide more combinations than a small set of very large experts. These are design motivations; the V4 paper does not separately measure every proposed benefit.

Expert specialization need not correspond to a clean human category such as “biology” or “Python.” Routing depends on learned hidden representations. An expert can participate in many contexts, and a task can use many experts across its tokens and layers.

Why active parameters are not a deployment size

Pro activates 49B parameters per token, while its total count is about 1.6T. Flash activates 13B out of 284B.

Unselected experts do not perform their main feed-forward arithmetic for that token. Their weights still exist. A server must hold them, distribute them, or fetch them when needed.

For a single token, the selected subset is small. For a large batch, the union of selected subsets can be much larger. The serving system must gather token groups for each expert efficiently enough to use matrix hardware well.

Small batches have another problem: an expert may receive very few tokens. Its matrix multiplication can become inefficient, and communication latency can dominate. V4's wave-based MegaMoE scheduling is partly aimed at these conditions.

Thus, MoE exchanges one problem for several smaller ones: routing, load balance, irregular batches, weight storage, and distributed communication.

The new routing score

The paper changes the affinity transformation from sigmoid to

Sigmoid approaches one for large positive inputs. The new transformation grows approximately as $\sqrt{x}$ when $x$ is strongly positive. For strongly negative inputs, it behaves approximately as $\exp(x/2)$.

This gives a different range and gradient profile to the routing scores. It is reasonable to expect a change in routing-weight behavior, but the paper does not provide a controlled V4 comparison proving a particular effect of this substitution.

There is a useful subtlety. Both sigmoid and square-root softplus are strictly increasing. If one selects top experts using only a vector of raw logits passed through either function, the ordering is unchanged. A difference must arise through such factors as weighting, normalization, balancing biases, or how training changes the logits.

Saying that a monotonic score function “finds different experts” in a fixed unmodified logit vector would therefore be misleading.

Why load balancing is necessary

A router could prefer a small set of experts. Those experts become busy while others remain idle. Training then suffers from both uneven learning and uneven hardware work.

V4 retains auxiliary-loss-free balancing and adds a small sequence-wise balance loss. The first approach adjusts expert-selection behavior through balancing biases rather than relying only on an auxiliary term added to the main training loss. The second helps prevent extreme imbalance within an individual sequence.

The reported bias update speed is 0.001. The sequence-level balance loss weight is 0.0001. These values describe the run; they are not a complete specification of the balancing algorithm.

The inherited V3 method makes the bias's role explicit. Selection uses the adjusted score $s_e+b_e$, while the contribution weight is derived from the original $s_e$:

This schematic omits any additional routed-output scaling. It separates the balancing decision from the weight assigned to an already selected expert.

An overloaded expert's bias decreases; an underloaded expert's bias increases. The bias is updated from observed load, rather than being only a gradient from an auxiliary balancing objective.

For an illustrative top-two choice, let affinities be $(0.8,0.7,0.2)$ and biases be $(-0.2,0,0.6)$. Selection sees $(0.6,0.7,0.8)$ and chooses experts two and three. Their normalized contribution weights use $0.7$ and $0.2$, giving approximately 0.778 and 0.222.

This also explains why a monotonic affinity transformation can matter despite preserving the unbiased ordering: adding fixed biases after the transformation depends on the scale and spacing of transformed scores.

The phrase auxiliary-loss-free names this balancing strategy. V4 still has a small sequence-wise auxiliary balance loss, so the complete objective is not free of every balancing loss.

Global and per-sequence balance are not identical. A batch can be globally balanced while one long sequence repeatedly selects the same experts. A small per-sequence control addresses that possibility, though it may also constrain natural specialization.

No balancing method makes every expert's workload identical at every instant. Serving kernels still need to handle irregularity.

Hash routing in the first three MoE layers

The initial three MoE layers route according to a predefined hash function based on input token ID. A repeated token ID follows the rule regardless of its later contextual meaning.

A hash rule is computationally simple and does not need a learned representation to make its selection. It also gives early layers a stable conditional path.

The paper does not fully disclose the hash construction in this section. We should not assume a particular number of hash tables, collision policy, or semantic grouping.

Hash collisions do not necessarily mean failure. The assigned expert processes multiple token types, just as a learned expert processes many hidden states. The hash partitions access to computation; it is not a direct dictionary of one expert per word.

Replacing the initial dense feed-forward layers with hash-routed MoE layers is a distinct change from adding mHC or compressing attention.

Removing the routing-node constraint

Earlier distributed designs can limit how many physical nodes a token's selected experts span. Such limits reduce communication complexity but constrain routing.

V4 removes the stated target-node constraint and redesigns parallelism to maintain efficiency. This gives routing more freedom at the cost of a harder systems problem.

The relevant consequence is conditional. It helps only if dispatch, expert computation, and combine can handle the resulting communication pattern. Chapter 12 explains the fused pipeline used to hide much of that communication.

Multi-token prediction

Ordinary language-model training predicts the next token. Multi-token prediction, MTP, adds objectives for future tokens through additional modules. V4 keeps the V3 strategy and uses depth one.

Depth one here means one additional prediction module; it does not mean the ordinary next-token objective is removed. The purpose is to provide additional learning signal about future sequence structure.

The inherited sequential design

V4 explicitly retains V3's MTP strategy. The local V3 source explains the mechanism more fully, so the following account is inherited detail rather than a newly disclosed V4 change.

At position $t$, the main model's hidden state $h_t$ predicts token $x_{t+1}$. An additional depth-one module combines that hidden state with the embedding of $x_{t+1}$ and predicts $x_{t+2}$.

In a simplified notation,

followed by a Transformer block and the shared output head:

The concatenated input has width $2d$ and is projected back to $d$. The embedding table and output head are shared with the main model. The additional Transformer block has its own computation.

During training, the observed next token is available from the training sequence. Giving it to the MTP module is not a causal leak into the ordinary next-token prediction: the module's target is the token after it.

For example, at the prefix “the result is,” the main objective predicts the next token. The MTP module then receives the observed next token and learns the following continuation. It is not asked to guess two mutually independent future tokens from the same unchanged representation.

With more MTP depths, this approach continues sequentially. V4 uses only one additional depth.

The extra loss

A schematic depth-one objective is

where both terms use cross-entropy on their respective targets. Valid positions and sequence boundaries determine where each target exists.

The extra objective encourages representations useful beyond one immediate continuation. It can densify the training signal. Those are motivations for MTP; they do not prove that every internal representation explicitly plans a future sentence.

V4 uses weight 0.3 for most pre-training and 0.1 when learning-rate decay begins.

Using MTP during generation

MTP can also support speculative decoding. A proposal mechanism produces candidate future tokens, and the target model checks them. Correct acceptance and correction rules are needed to preserve the target distribution.

At inference, the main model first produces a candidate next token. The MTP module can use that candidate's embedding to propose a following token. Verification determines which proposals can be accepted.

The module can also be omitted while the main model generates normally. Its training contribution does not require every deployment to use speculative decoding.

Speculation is worthwhile when verification saves more work than proposal and correction add. High agreement, efficient multi-token verification, and suitable batching help. Low agreement or an expensive proposal path can erase the benefit.

The V4 report does not provide a full serving acceptance-rate study or a universal MTP speedup. MTP depth does not establish how many tokens an application will produce in one target-model pass.

A token-level example

Imagine two tokens at a later learned-routing layer. One occurs in a mathematical proof and another in a shell error log. Their hidden vectors differ because prior attention and transformations have incorporated context.

The router scores all routed experts for each vector, applies the model's selection rule, and picks six. The tokens may share some selected experts and differ on others. Both also use the shared expert.

A distributed runtime groups the tokens by their target experts, sends the appropriate activations, computes the expert outputs, and returns weighted results to the originating token positions.

The token does not leave the model for six independent reasoning agents. It receives six feed-forward transformations within one layer. The same process repeats in later layers.

This is why internal MoE routing and external agent orchestration should be kept separate in explanations of V4.

Source: local V4 main.tex, Designs Inherited from DeepSeek-V3, lines 290–305; model and training setups, lines 927–985. Inherited balancing and sequential MTP details are expanded from local papers/dsv3/main.tex, lines 358–384 and 410–457. Expert size, score-function behavior, and the routing example above are derivations.

Continue with CSA compression or return to the contents.

5. How CSA Compresses Memory

Compressed Sparse Attention has two steps. It first builds a shorter memory sequence. It then chooses entries from that sequence for each query. This chapter explains the first step.

Compression is learned pooling over hidden states. It is not a text summarizer that writes a shorter paragraph, and it is not a lossless file-compression algorithm.

The input and the four projections

Let

be the input hidden states. CSA forms two candidate KV sequences and two sequences of compression logits:

Each result has shape $N\times c$, where $c=512$ in both reported V4 sizes.

The $C$ projections produce content to be stored. The $Z$ projections produce learned preferences for pooling that content. All four matrices have shape $d\times c$.

Two learned positional-bias matrices, $B^a$ and $B^b$, each have shape $m\times c$. These let the compression rule distinguish positions within a block.

One weight distribution per coordinate

The compressor does not assign one scalar weight to an entire token vector. It assigns a weight separately for each coordinate.

For a coordinate $r$, it collects compression logits from the relevant $2m$ source positions, adds positional biases, and performs a softmax across those positions. The weights for coordinate $r$ sum to one.

Different coordinates can prefer different tokens. One coordinate can emphasize the beginning of the block while another emphasizes a later position.

This gives the compressor more flexibility than a uniform average. It still produces only one $c$-dimensional entry, so it cannot generally preserve all independent information from $2m$ arbitrary $c$-dimensional vectors.

Which positions enter a compressed block

Using zero-based indexing, compressed entry $i$ draws from:

• the $a$ branch of the current block, positions $mi$ through $m(i+1)-1$;
• the $b$ branch of the previous block, positions $m(i-1)$ through $mi-1$.

For coordinate $r$, define the source logit

Softmax gives weights

The stored coordinate is

The paper calls the normalization a softmax along the row dimension. The intended operation is across source positions, independently for each coordinate. It is not softmax across the 512 coordinates of one token.

A scalar worked example

Reduce the example to one coordinate and use $m=2$. Suppose the two current-block candidate values are 10 and 20, and the two previous-block candidate values are 2 and 6.

After adding biases, suppose the four compression logits are

Their exponentials are $(2,1,1,1)$, giving weights

The compressed coordinate is

A second coordinate can use a completely different set of weights. The final vector is built from these coordinate-wise pooled values.

The example also shows what compression does not guarantee. The value 9.6 does not uniquely identify the original four values. A later query reads a learned representation useful for prediction, not a reversible encoding of each source coordinate.

Why eight source tokens still means four-to-one compression

With the reported $m=4$, each interior entry draws from eight source token positions. Yet one new entry is created per four new original tokens.

An original block appears in two neighboring summaries through different learned projections. The number of summaries still grows as $N/4$.

Confusing receptive field with output stride leads to the wrong cache estimate. The receptive field is eight tokens for an interior entry. The stride is four tokens. The stride determines long-run entry count.

At the start of a sequence there is no previous block. The missing branch uses zero content and negative-infinite logits, so it contributes no softmax weight.

What overlap may help

Fixed blocks introduce arbitrary boundaries. A short phrase can straddle the end of one block and the start of another.

Overlap gives a summary access to both sides of that boundary. It may help preserve short dependencies that a disjoint compressor would separate.

The two projections matter. The previous block is not merely copied into the next summary. Its $b$ representation can differ from its $a$ representation.

This is a reason for the design, not a claim that the V4 report isolates overlap's effect on every long-context task. A controlled test would compare overlapping and non-overlapping compression at matched capacity, training, and cost.

When an entry becomes available

Causal language modeling cannot let a query use a compressed entry containing future tokens. A summary of positions 4–7 cannot be read by the query at position 5.

The paper restricts each query to preceding completed compressed blocks. Its indexer condition is

Here $s$ is a compressed-block index and $t$ is the query's original token position under the paper's indexing convention.

Implementation details at a block-completion boundary require consistent indexing. The safe conceptual rule is simple: no summary may expose information from an original position beyond the query's causal horizon.

The same rule must apply to selection scores. Masking the final attention but allowing future blocks into the indexer's top-k computation would still create a dependency on future information.

Why the local window is necessary

Before a block is completed, its tokens have not produced a usable compressed entry. Even after completion, the design avoids treating the current block as ordinary prior compressed history.

A local uncompressed window gives the query access to recent tokens directly. In V4 its width is 128, much larger than CSA's four-token block.

The local window therefore serves two purposes: preserving fine recent detail and covering the information gap created by block compression and causality.

It does not restore uncompressed access to arbitrarily old text. Once information leaves the local window, the compressed paths carry it.

What is retained during decode

An inference server needs state for tokens that have arrived but cannot yet be finalized into a compressed entry. Overlap also requires recent branch information for the neighboring block.

The paper therefore separates a growing classical cache of completed compressed entries from bounded state for local attention and compression tails.

These buffers are part of the architecture's real memory cost. A formula that counts only completed compressed entries underestimates short-context and per-request memory.

The main cache grows with $N/4$. Tail state is bounded by the compression mechanism rather than by the entire sequence.

How the compressor learns

The projection matrices and positional biases are trainable. They receive gradients through the model's training objectives where the computation permits them.

The paper does not introduce a separate reconstruction requirement that forces a compressed entry to recover every original token. It trains a model for language prediction and later task behavior.

It is therefore sensible to describe the compressor as learning task-useful memory. It is not justified to claim that it retains every fact or that each coordinate has a known semantic meaning.

The hard selection step in CSA introduces additional training questions. V4 describes an indexer warmup and a two-stage introduction of sparsity but does not fully restate the indexer's supervision recipe. That gap is discussed in Chapter 6.

Source: local main.tex, Compressed Key-Value Entries under CSA, lines 389–407; causal and local-attention discussion, lines 515–521; cache management, lines 831–875. The scalar example and the distinction between receptive field and stride are explanatory derivations.

Continue with CSA selection and reading or return to the contents.

6. How CSA Selects and Reads Memory

CSA first compresses history into roughly one entry per four original tokens. It then uses a smaller attention-like mechanism to choose which entries the main attention should read.

The selector is called the lightning indexer. Its job is to rank compressed blocks cheaply. The main attention's job is to read the selected content using a richer set of query heads.

Why selection needs its own representation

A million-token context produces roughly 250,000 CSA entries. Reading every 512-dimensional entry with every main query head would still be expensive.

The indexer uses 128-dimensional keys and 64 query heads. Its compressed keys are built with the same kind of compression as the main CSA memory, but they have separate learned content.

“Same compression operation” does not mean the indexer keys are a slice of the main KV vector. The dimensions and learned projections serve a different task.

Low-rank query generation

For token $t$, the model first creates a query latent:

This maps the hidden vector from width $d$ to width $d_c$. Flash uses $d_c=1024$; Pro uses $d_c=1536$.

The latent is then expanded into indexer queries:

The result contains 64 heads of dimension 128. Main attention queries are also produced from $c_t^Q$, through a separate projection.

Sharing this latent saves duplicated down-projection work. It also couples the representations used to propose memory entries and read them. It does not force the indexer and main attention to have identical scores.

The index score

The model creates a learned weight for each indexer head:

For a preceding compressed block $s$, the score is

Each query head computes a match with the shared indexer key. ReLU clips negative dot products to zero. The head weights combine these matches into one scalar ranking score per compressed block.

The displayed formula gives the head weights as a linear projection without a positivity constraint. We should not assume all $w_{t,h}^I$ are positive merely because the dot products pass through ReLU.

The score is not a main-attention probability. It need not sum to one across blocks. It is used to select a set.

A small ranking example

Suppose an illustrative two-head indexer gives nonnegative matches:

If the query's head weights are $(0.8,0.2)$, the aggregate scores are

A top-two selector chooses A and C. If the weights change to $(0.2,0.8)$, the ranking changes and B becomes most preferred.

The main attention may then place different weights on the selected blocks. Selection decides which entries are available to read; main attention decides their contribution.

Top-k is a budget over compressed entries

Flash selects up to 512 compressed entries per query. Pro selects up to 1,024.

These values are not counts of original tokens. With stride four, 1,024 selected entries represent 1,024 learned summaries. Their source receptive fields can overlap. Multiplying by four or eight does not produce a set of individually accessible raw tokens.

The selected entries need not be adjacent. They can come from distant parts of the sequence. All main query heads use the selected compressed-memory set for that token.

When the eligible history has fewer than $k$ entries, the selector can retain all of it. At longer lengths, the budget becomes binding.

The sliding window is an additional source of entries. It does not consume the reported compressed-entry top-k budget.

Selection reduces reading, not all scanning

For one query, the indexer still scores the eligible compressed candidate sequence. Its approximate arithmetic grows as

Main CSA attention grows approximately as

once the top-k budget is filled.

This distinction matters. CSA caps the number of main-memory reads but retains a length-dependent selection cost. The indexer uses a smaller dimension and FP4 computation to make that scan cheaper.

For prefill, many queries scan different eligible prefixes. With a fixed compression factor, the indexer's pairwise work remains quadratic in original sequence length, with a much smaller coefficient. It is inaccurate to describe the entire CSA layer as constant-time decode or linear-time prefill solely because $k$ is fixed.

Shared key–value MQA

The main query heads are generated as

For each head, the selected compressed entry serves as both key and value:

This makes two sharing decisions. All query heads share the main memory, and each memory vector plays both the matching and content roles.

Ordinary MQA often shares KV heads but still uses separate key and value vectors. V4 ties these roles at the stored-entry level, making its payload smaller again.

The same vector can still participate in different computations. A dot product with the query determines a score. A weighted sum carries the vector into the output.

Normalization, position rotation, local entries, and sink logits modify this simplified expression. Chapter 7 explains those details.

Why output projection is a large cost

The main attention outputs are wide. Flash produces $64\times512=32768$ coordinates. Pro produces $128\times512=65536$.

A direct projection back to hidden width $d$ would use a matrix with $cn_hd$ weights.

V4 groups the heads. Each group first projects its own outputs to an intermediate width $d_g=1024$. The intermediate group outputs are concatenated and projected to $d$.

The principal weight count becomes

For Flash, the direct and grouped counts are:

The grouped structure halves this part's principal parameter count.

For Pro:

That is about 39.3% of the direct count, a reduction of about 60.7%. These are derived counts for the projection matrices, excluding other attention weights.

What grouped projection gives up

A direct linear map can assign arbitrary weights from each head coordinate to each hidden output coordinate. Grouped projection inserts a narrower intermediate representation within each group.

With no intervening nonlinearity in the described projection, the combined operation is still linear, but its factorization constrains which linear maps can be expressed. The intermediate bottleneck is the source of the saving.

For each group, 4,096 input coordinates become 1,024 coordinates. Both Flash and Pro use eight heads per group, so this local compression ratio is the same.

The architecture accepts that restriction to reduce compute. The paper's full-model results show the chosen structure is workable, but do not isolate how much quality is lost or gained by the factorization.

How a hard selector can be trained

Top-k selection is discrete. The chosen indices do not provide an ordinary smooth derivative describing how the selected set changes.

Gradients can train the content of selected memory entries and the main attention. Training the indexer's ranking requires a suitable objective or method.

The V4 source says it first warms up the model with dense attention, uses a short lightning-indexer warmup, then trains with sparse attention. It refers to DeepSeek Sparse Attention and earlier settings. It does not fully restate the indexer's target distribution, loss, or gradient isolation in this section.

We should therefore avoid inventing a complete indexer training loss from the forward-pass formula. A reproduction would need the inherited method and training code.

Two independent failure points

If the compressor discards a crucial detail, a perfect selector cannot recover it. If the detail survives compression but the selector omits its entry, the main attention cannot use it through that CSA path.

Another layer's HCA path or the remaining local window may provide information. This gives the full model multiple opportunities to use history. It does not make individual CSA selection errors harmless.

The right empirical questions are whether relevant entries remain findable, whether downstream answers use them correctly, and how these properties change with length and distraction.

Source: local main.tex, Lightning Indexer, Shared Key-Value MQA, and Grouped Output Projection, lines 409–449; sparse-training setup, lines 961–985. Projection counts and complexity expressions are derived.

Continue with HCA and positional details or return to the contents.

7. HCA, Local Attention, and Position

HCA stores one main memory entry per 128 original tokens and reads all eligible entries. It trades fine sequence resolution for broad access to a short memory.

Both HCA and CSA also include a local uncompressed window, query and KV normalization, partial RoPE, and attention sinks. These details are necessary to understand what the main formulas leave out.

HCA's compressor

Given hidden states $H\in\mathbb{R}^{N\times d}$, HCA computes

The matrices produce content and compression logits, each with shape $N\times c$. A learned bias $B\in\mathbb{R}^{m'\times c}$ distinguishes positions within a block.

For each block and each coordinate, softmax normalizes $m'=128$ logits across the block's source positions. The compressed vector is the weighted sum:

HCA has one content branch and one logit branch. It does not use CSA's two-block overlap.

The output sequence length is roughly $N/128$. At one million tokens, the number of complete blocks is 7,812 with 64 tokens remaining under a decimal million-token convention.

Dense attention over compressed history

HCA generates multiple queries through a low-rank query latent, then performs shared key–value MQA over all eligible compressed entries.

It also uses grouped output projections. The main query head counts and dimensions are the same as in CSA for a given model size.

The word dense refers to reading all entries in this compressed memory. It does not mean reading every original token independently.

For decode, the main HCA history cost grows approximately as $N/128$. For full-sequence prefill, the pairwise part still grows quadratically with $N$, reduced by the compression factor. A fixed compression ratio changes the coefficient, not the asymptotic exponent.

Why use HCA when CSA reads fewer entries?

At a million tokens, Pro's CSA reads at most 1,024 main compressed entries while HCA reads about 7,812. HCA can therefore do more main-attention work per query.

Its advantages are elsewhere. It stores much less history and avoids a separate top-k indexer. It gives the query broad access to every coarse historical block rather than asking a selector to omit most blocks.

CSA and HCA make different trade-offs:

Interleaving them lets later representations be influenced by both kinds of access. A tempting explanation is that HCA supplies broad context and CSA supplies selected detail. That is a useful interpretation of the design, not a measured division of labor proven in the report.

The sliding window

Both attention types use up to 128 recent uncompressed KV entries alongside compressed entries.

The window helps with language's frequent dependence on nearby text. It also covers unfinished blocks whose summaries are not yet available.

For a query in the middle of a 128-token HCA block, the window provides recent information that HCA cannot expose through that current block's completed summary.

“Uncompressed” here means not pooled across several sequence positions. The local entries are still projected learned vectors. They are not raw text.

The paper describes combining local entries with compressed entries in core attention. It should not be pictured as an independent full attention module whose output is necessarily added with a fixed hand-chosen weight.

Does information appear twice?

Recent tokens can be represented in a completed compressed entry and also in the local window. CSA overlap can make an original block influence neighboring summaries as well.

Such repeated representation is not inherently invalid. The memory entries have different roles and learned content. Attention can learn how to use them.

It does mean that simple counts of “tokens covered” are misleading. A set of selected compressed entries can have overlapping source spans. The model is mixing learned representations, not performing an exact union of unique original words.

For implementation, the important rule is that every included representation obeys the causal mask.

Query and KV normalization

Before core attention, V4 applies RMSNorm separately to each query head and to the shared compressed KV head.

Dot-product logits can grow when vector magnitudes grow. Normalization controls that source of logit growth, helping avoid extremely concentrated softmax distributions and numerical instability.

A normalized vector still carries direction and relative coordinate patterns. The norm operation does not erase the information used for matching.

The paper uses this design to explain why its Muon setup does not need QK-Clip. That is a specific reported choice, not a general claim that query normalization always removes every attention instability.

Partial rotary position embeddings

V4 applies RoPE to the last 64 coordinates of queries and KV entries. The other 448 coordinates of a 512-dimensional main vector remain non-rotary.

Let $R_t$ denote the rotation for position $t$. On the rotary subspace, a query and key score includes

This follows from orthogonal rotations and the composition rule for RoPE. The score depends on relative position.

The paper does not fully specify every compressed-entry position convention in this section. For a correct implementation, a block must be assigned a consistent rotary position and a consistent causal interpretation.

Why the output is de-rotated

V4's stored vector is both key and value. Rotating a key therefore also rotates the content read as a value.

Suppose the rotary part of the head output is

Without correction, the output includes absolute-position rotations from the source entries. V4 applies the inverse query-position rotation:

The output contributions now carry relative rotations with respect to the query. This operation concerns the rotary coordinates only.

De-rotation does not undo compression, recover individual tokens, or reverse the attention weights. It changes the positional basis of the combined value representation.

The derivation assumes the standard rotation composition used by RoPE. Exact frequency scaling and compressed-position choices belong to the implementation.

Attention sinks

Ordinary softmax makes the weights over real entries sum to one. A head must distribute all of its attention mass somewhere, even if none of the available entries is useful.

V4 adds a learned sink logit $z'_h$ to the denominator:

The real-entry weights now sum to at most one. The remaining mass can be interpreted as going to a zero-valued extra entry.

For example, suppose the real entries have total exponential mass 4 and the sink has mass 6. The real weights sum to $4/(4+6)=0.4$. The resulting value mixture is reduced accordingly.

This lets a head read very little when appropriate. It does not let the entire Transformer skip a layer, since projections, residual operations, and other heads still run.

The sink is one learned logit per head in the description. It is not a retrieved document or an original token that the server must append to the prompt.

What a million-token claim leaves open

A million-token context remains available only through the representations the architecture keeps. Old exact detail can be lost through pooling. CSA can miss a surviving detail through selection. HCA can blur many facts into one coarse entry.

Several layers and two attention types can reduce the practical impact of these failures. They do not provide a theorem of perfect recall.

The report's long-context measurements are therefore necessary evidence. They assess whether the compression choices preserve enough information for the evaluated tasks. Chapter 18 examines those results without equating them with arbitrary lossless retrieval.

Source: local main.tex, HCA, lines 461–496; Other Details, lines 497–531. Positional algebra, the sink example, and complexity statements are derived explanations.

Continue with the efficiency model or return to the contents.

8. A Cost Model for Long Context

V4 reduces several different costs: stored history, main-attention reads, selection arithmetic, output projection, and expert-weight traffic. These savings have different units and different limits.

This chapter builds a simple accounting model. Its calculations are estimates based on reported dimensions. They are not hardware measurements or a reconstruction of the paper's complete FLOP figure.

Separate the memory categories

An inference server uses memory for weights, per-request state, and temporary work.

Weights are largely fixed for a loaded model. KV caches grow with the number and lengths of active requests. Compression-tail and local-window state are bounded per request. Workspaces depend on batch shapes and kernels.

A model can fit its weights and still lack enough room for many long contexts. It can also have a small cache and still be impractical because its total weights require a large cluster.

A conventional cache baseline

For BF16 GQA with eight KV heads and head dimension 128, one layer stores

bytes per original token. The factors are separate K and V, head count, head dimension, and bytes per scalar.

At one million tokens, this is 4.096 GB per layer. Across 61 layers, the payload is 249.856 GB. Across 43 layers, it is 176.128 GB.

This matches the type of baseline used in the paper's discussion. It is a dimensional comparison, not a claim that a specific competitor has exactly that number of layers and configuration.

The main compressed-entry payload

V4 stores the last 64 rotary dimensions in BF16 and the other 448 dimensions in FP8. One 512-dimensional shared KV entry therefore has payload

Separate K and V copies are not required for the main tied representation.

A CSA layer creates approximately one main entry per four original tokens. Its main growing payload is therefore about 144 bytes per original token.

An HCA layer creates one main entry per 128 original tokens. Its corresponding cost is about 4.5 bytes per original token.

CSA also stores indexer keys. If a 128-dimensional indexer key is stored entirely in FP4, its scalar payload is 64 bytes, before scale factors and padding. This adds about 16 bytes per original token.

Under these assumptions, a CSA layer's growing scalar payload is approximately 160 bytes per original token.

A whole-model approximation

Interpreting the reported alternation literally gives 30 CSA and 31 HCA layers for Pro, and 21 CSA, 20 HCA, and two pure-window layers for Flash.

The growing cache payload is then approximately

and

At one million tokens, these estimates are about 4.94 GB and 3.45 GB. They exclude MTP modules, scale metadata, alignment, allocator overhead, and bounded state.

Compared with the layer-matched GQA8 payloads above, both are close to 2%. This gives a concrete explanation of the paper's approximate 2% cache claim.

It does not establish an actual runtime allocation of exactly 4.94 GB for Pro. A serving engine may reserve blocks in advance or use different representations and padding.

Why scales and tails matter

Low-precision formats usually store scale factors alongside quantized values. Small blocks need more scales per value than large blocks.

An FP4 key with 128 values has 64 bytes of scalar payload. Four one-byte scales for groups of 32 would add four bytes if that exact format is used. Other bookkeeping can add more.

Compression also needs unfinished-block state. Overlapping CSA may retain branch projections for neighboring blocks. HCA retains information until a 128-token block is ready.

At very long contexts, the growing compressed cache dominates these bounded buffers. At short contexts, the buffers and allocation granularity can be a large fraction of total per-request state.

This is why a long-run bytes-per-token formula should not be applied blindly to short requests.

Main-attention arithmetic

For one query head reading $E$ entries of dimension $c$, the dot products use roughly $2cE$ FLOPs and the weighted value sum uses roughly another $2cE$. The pairwise part is approximately

This excludes projections, normalization, softmax, sink handling, and kernel overhead.

For Pro CSA after its top-k budget is filled,

With 128 heads and $c=512$, this gives roughly 302 million FLOPs per layer per query for the main pairwise operations.

For Pro HCA at one million tokens,

giving roughly 2.08 billion FLOPs per layer per query.

HCA can do more main reading than CSA even while storing much less memory.

Indexer arithmetic

The CSA indexer scores roughly $N/4$ candidate entries. Ignoring ReLU and aggregation overhead, its dot-product work is approximately

With 64 heads, dimension 128, and $N=10^6$, that is about 4.096 billion scalar FLOPs per CSA layer per query.

This is larger than the simplified Pro CSA main pairwise count. The indexer uses FP4 computation, smaller vectors, and specialized kernels, so scalar FLOP counts alone do not determine elapsed time.

The calculation explains why optimizing the indexer matters. It also explains why reducing $k$ cannot eliminate the full long-context cost.

What happens during prefill

During prefill, queries appear at many positions and have different eligible prefix lengths.

HCA reads a compressed prefix of length approximately $t/128$ at position $t$. Summing across $t$ gives pairwise work proportional to $N^2/128$.

CSA's indexer similarly scans compressed prefixes of length approximately $t/4$. Its selection work remains proportional to $N^2/4$, with small head dimensions and low precision.

CSA's main attention has bounded reads per query after the selected-memory budget fills. That part is approximately linear in $N$ for fixed $k$, while short prefixes may read all available entries.

Thus the complete hybrid design is much cheaper than full-resolution dense attention, but it is not asymptotically linear in all prefill attention work at fixed compression factors.

The reported V3.2 comparison

At 1M context, the paper reports Pro at 27% of V3.2's single-token inference FLOPs and 10% of its KV cache. Flash is at 10% and 7%.

These numbers include more than the simplified terms above. They involve the earlier model's attention, different active parameter counts, and the report's equivalent-FP8 accounting.

The GQA8 2% claim and the V3.2 10% claim use different baselines. They are not inconsistent.

The paper also notes that current hardware gives FP4-by-FP8 operations the same peak FLOPs as FP8-by-FP8, while a future implementation could make them one-third more efficient. This is a proposed hardware opportunity, not a measured V4 serving speedup.

Why FLOPs do not imply a latency ratio

Latency depends on where work is limited.

A long attention scan can be limited by memory reads or selector throughput. A small MoE expert batch can be limited by launch and communication latency. A wide matrix multiplication can be limited by arithmetic throughput. A fused kernel can hit power limits while activating several resources at once.

Cache layout and reuse also matter. A shared prefix loaded from disk can avoid a large prefill. A cache miss can make an otherwise cheap decode design feel slow.

Therefore “27% of FLOPs” does not mean “27% of response time.” A valid latency comparison must specify hardware, precision, batch size, sequence lengths, cache-hit rate, and scheduling.

A useful performance worksheet

For a real deployment, record:

A report that gives only average tokens per second cannot answer all these questions.

Source: local main.tex, efficiency claims, lines 217–227 and 533–545; dimensions, lines 927–957; FP4 discussion, lines 1246–1264. All byte and operation calculations in this chapter are derived estimates with the stated exclusions.

Continue with mHC or return to the contents.

9. Manifold-Constrained Hyper-Connections

Ordinary residual connections carry one vector per token between layers. Manifold-constrained hyper-connections, mHC, carry several vectors and learn how information moves among them.

V4 uses four residual streams. The costly attention or MoE sublayer still processes a vector with the normal hidden width. The extra width belongs to the residual state.

From one stream to several

A conventional update is

With $n$ streams, the residual state is

The generalized update is

The mappings have different jobs:

$A_lX_l$ has width $d$, so $F_l$ does not become four times wider.

For $n=1$ and mappings fixed appropriately to one, this structure reduces to the familiar residual update.

Why unconstrained mixing can be unstable

Along the direct residual path, repeated mixing gives products such as

If these matrices repeatedly expand some direction, the product can amplify it greatly. They can also attenuate or cancel signals unpredictably.

This resembles multiplying many gains in a deep system. A small excess per layer can become a large effect across a stack.

The paper reports numerical instability in naive hyper-connections and constrains $B_l$ to reduce this problem.

Doubly stochastic matrices

The allowed matrix has nonnegative entries and every row and column sums to one:

The set is called the Birkhoff polytope. The paper uses “manifold” in naming the method; mathematically, the feasible set is a convex polytope, with a boundary.

Row sums ensure each output stream is a convex mixture of input streams on the residual path. Column sums preserve the total stream sum under that mixing.

For one hidden coordinate across four streams, no output coordinate can lie outside the minimum and maximum of the four input values solely because of $B$.

A four-stream example

Consider

Every row and column sums to one. If one hidden coordinate across the streams is $(10,0,4,2)$, residual mixing produces $(7,3,3.2,2.8)$.

The sum remains 16. The first pair becomes less different, and the second pair also becomes less different.

This example is intentionally simple. A learned matrix can mix every pair of streams, and it can depend on the current input.

Why the spectral norm is bounded

For a nonnegative doubly stochastic matrix,

The standard norm inequality gives

Thus the direct linear residual mixing is non-expansive in Euclidean norm. Since $B\mathbf1=\mathbf1$, there is also a preserved direction, so its largest singular value is one for an exactly doubly stochastic matrix.

The set is closed under multiplication. If $B$ and $D$ satisfy the constraints, so does $BD$. A long product therefore remains non-expansive along this direct linear path.

This is the core mathematical reason for the constraint.

What the bound does not prove

The full update includes the nonlinear sublayer and input-dependent mappings:

Its Jacobian contains derivatives of $A$, $B$, $C$, and $F$. The norm bound on the realized $B$ does not bound all these terms.

Even the derivative of $B(X)X$ includes a term from changes in $B$ with $X$. Holding $B$ fixed gives one stability argument; differentiating the complete dynamic map is a broader problem.

Therefore mHC's constraint does not prove that the entire Transformer is non-expansive or immune to exploding gradients. It controls a specific source of repeated amplification and is supported by the authors' training experience.

The distinction matters because the model still needed MoE stability interventions.

Input and output constraints

The paper sets

Each input coefficient lies between zero and one. Each output coefficient lies between zero and two.

These mappings are nonnegative and bounded, but they are not normalized to sum to one. The input combination can add several streams together, and the output can inject the sublayer result into several streams.

A negative value in the residual vectors can still combine with a positive value. Nonnegative coefficients avoid cancellation caused by signed mixing weights; they do not eliminate every possible cancellation between values.

Dynamic and static components

The flattened residual state is normalized:

The raw mappings combine input-dependent projections with learned static biases:

The $\alpha$ values are learned gates initialized small. This initially limits the size of input-dependent changes. The static terms provide a baseline pattern.

For four streams, the three mappings require $4+16+4=24$ raw coefficients. The projection takes the flattened four-stream state and generates this small output.

The mapping is cheap relative to large expert and attention matrices, but the expanded activations still affect memory and communication.

Sinkhorn–Knopp normalization

The raw residual matrix is exponentiated to make its entries positive:

Then the algorithm alternates column and row normalization:

Under the usual positive-matrix conditions, repeated normalization converges toward a doubly stochastic matrix. V4 uses 20 iterations.

For a small example, exponentiating a $2\times2$ raw matrix can give

Normalizing either rows or columns yields

which already satisfies both constraints.

A general matrix needs repeated steps. Twenty finite iterations produce a practical approximation. Exact constraints and exact norm claims belong to the ideal limit; floating-point and finite-iteration errors should be checked when validating an implementation.

Calling this a projection is convenient, but it does not imply ordinary Euclidean nearest-point projection onto the polytope.

The engineering cost

Four residual streams increase activation memory and pipeline communication. V4 uses fused mHC kernels, selective recomputation, and a modified DualPipe overlapping schedule.

The paper reports 6.7% wall-time overhead within the overlapped 1F1B pipeline stage after these optimizations. This is a specific training-stage measurement, not a universal overhead for every inference batch or model.

Static biases, gates, and RMSNorm weights use AdamW. Most matrix parameters, including the dynamic mHC projections, use Muon.

What an ablation would clarify

To separate the effect of mHC from other V4 changes, one would compare matched models with ordinary residuals, unconstrained hyper-connections, and mHC. Training stability, quality, activation memory, and wall time would all matter.

The dedicated mHC paper provides a separate line of evidence. The V4 report summarizes the method and its integration. It does not turn the full V4 benchmark gain into an isolated estimate of mHC's contribution.

Source: local main.tex, mHC, lines 306–363; efficient implementation, lines 776–788. The norm proof, Jacobian qualification, and small matrix examples are derived explanations.

Continue with Muon or return to the contents.

10. Muon and Matrix-Shaped Updates

Muon treats a weight matrix as a matrix when forming its update. AdamW mainly adjusts parameter coordinates using running gradient statistics. This difference is the starting point for understanding V4's optimizer.

The paper reports faster convergence and greater stability with Muon. It provides the update algorithm and explains how it handles distributed training. It does not provide a complete matched V4 training-curve comparison that lets us assign every gain to the optimizer.

The update sequence

For one logically independent matrix $W\in\mathbb{R}^{n\times m}$, the reported algorithm is

$M_t$ is the momentum buffer. The additional $\mu M_t+G_t$ is the Nesterov-style input to orthogonalization. Weight decay is applied through the multiplicative factor on the old weight.

V4 sets $\mu=0.95$, $\lambda=0.1$, and targets an update RMS of 0.18 through the rescaling.

The normalization convention for momentum differs across optimizer implementations. This chapter follows the displayed V4 algorithm.

What orthogonalizing an update means

Let a momentum-like matrix have singular value decomposition

The idealized orthogonalized form keeps the singular vectors while replacing positive singular values with one:

where $r$ is the rank and the matrices contain the corresponding singular vectors.

This is a partial isometry for a rank-deficient matrix. A rectangular matrix cannot have both orthonormal rows and orthonormal columns unless the dimensions permit it.

The operation changes the geometry of the update. A very large singular value does not remain proportionally larger than a small positive one. Directions represented by different singular modes receive a more balanced scale.

This is not ordinary clipping of each matrix entry, and it is not normalization of the whole matrix to unit Frobenius norm. Those operations preserve different information.

A two-dimensional example

Take

Its normalized Frobenius form is approximately

The ratio of the two directions remains ten to one.

Its ideal polar factor is the identity matrix. Both directions have singular value one.

Real updates need not be diagonal. Singular vectors encode combinations of coordinates. Muon acts on those matrix directions rather than independently adjusting each scalar.

There is also a trade-off: equalizing modes can amplify a weak or noisy direction relative to a strong one. The optimizer's behavior depends on momentum, scale, approximation, and the training problem.

Why not use a full SVD?

A full singular value decomposition of every large weight matrix at every training step is expensive. Muon approximates the relevant transformation with matrix multiplications.

First normalize the input:

This bounds the maximum singular value by one, aside from numerical details and the need to handle a zero matrix.

One Newton–Schulz-style iteration is

Matrix multiplication hardware can execute these operations efficiently, especially when matrices of the same shape are batched.

An implementation can choose an orientation that makes the square product smaller. The V4 report gives the mathematical form, not every kernel scheduling choice.

The iteration acts on singular values

Because the update is a polynomial in $M$ and its Gram matrix, a singular value $\sigma$ is transformed by

The singular vectors remain the same in exact arithmetic. The iteration changes the singular values.

V4 uses ten steps. The first eight use

The final two use

The authors describe the first stage as moving singular values quickly toward one and the second as stabilizing them near one.

For the final polynomial, $f(1)=1$ and $f'(1)=0$. That makes one a stationary fixed point locally. Finite iterations and finite precision still produce an approximation rather than an exact SVD polar factor.

A zero singular value remains zero under the polynomial. Rank deficiency is not repaired by creating a new singular direction.

Why multiply by the square root of the larger dimension?

For a full-rank $n\times m$ matrix, the ideal orthogonalized update has $\min(n,m)$ singular values equal to one. Its squared Frobenius norm is therefore $\min(n,m)$.

Its per-entry RMS is

Multiplying by $\sqrt{\max(n,m)}\gamma$ gives RMS $\gamma$.

This lets the training setup reuse an AdamW-like learning-rate scale. In V4, the chosen target is 0.18.

The equality assumes full rank and ideal orthogonalization. A rank-deficient or approximate update can differ. The scaling is a practical convention, not a guarantee that every step has exactly the same RMS.

Which parameters use AdamW?

The architecture section explicitly retains AdamW for:

• the embedding module;
• the prediction head;
• mHC's static biases and gating factors;
• all RMSNorm weights.

All other modules use Muon in that description.

The later training-setup paragraphs list embedding, prediction head, and RMSNorm, but do not repeat the mHC exception. Reading the sections together avoids treating the shorter list as a contradiction that silently moves biases to Muon.

Embeddings and output heads have special roles, and many norm or bias parameters are not ordinary two-dimensional hidden-layer matrices. The report follows the established practice of using a conventional optimizer for these cases.

AdamW's reported settings are $\beta_1=0.9$, $\beta_2=0.95$, $\epsilon=10^{-20}$, and weight decay 0.1.

Why query normalization matters here

A matrix-shaped optimizer changes how projection weights evolve. Attention can become unstable if query and key magnitudes make logits excessively large.

V4 applies RMSNorm directly to attention queries and KV entries, and therefore does not use QK-Clip in its Muon configuration.

This is a compatibility decision between architecture and optimization. It is not evidence that Muon requires one universal normalization scheme.

The distributed-training consequence

AdamW's core update can be computed for a shard of scalar parameters using corresponding gradient and state shards.

Muon needs the complete logical matrix to form its Gram products and orthogonalized update. Splitting a matrix into arbitrary pieces and orthogonalizing each piece generally produces a different result.

V4 therefore shards whole matrices across ranks where possible. Expert matrices are naturally numerous and independent, making them convenient units. Dense matrices need a more constrained assignment.

This is an example of an algorithm shaping the systems design. The optimizer cannot be dropped into a scalar-sharded implementation without changing how updates are assembled.

What remains unknown from the V4 report

The source does not supply enough controlled evidence to quantify the optimizer's independent contribution to final benchmark quality. Faster convergence also needs a cost definition: fewer steps, fewer tokens, or less elapsed time can lead to different conclusions when an optimizer has additional matrix work.

A useful comparison would match architecture, data order, precision, learning-rate tuning, and compute budget. It would report stability events and elapsed time alongside loss.

The algorithm is clear enough to understand the proposed update. The full empirical attribution is less complete.

Source: local main.tex, Muon algorithm and discussion, lines 546–593; optimizer settings, lines 961–985; distributed implementation, lines 755–774. Singular-value and RMS derivations are explanations developed for this book.

Continue with pre-training and stability or return to the contents.

11. Pre-Training and Stability

Pre-training produces the base model from a large corpus. V4-Flash is trained on 32T tokens and V4-Pro on 33T. The paper describes data priorities, schedules, and two interventions for training instability.

The report is detailed about several hyperparameters but does not disclose a complete reproducible data and compute budget. Understanding what is stated and what is missing is part of reading the recipe.

The corpus

The data builds on DeepSeek-V3's corpus and processing pipeline. The authors emphasize mathematical and programming material, web text, multilingual data, and long documents such as scientific papers and technical reports.

For web data, they filter batched auto-generated and templated material. Their stated motivation is to reduce the risk of model collapse from poor synthetic or repetitive data.

The source also says agentic data is added during mid-training to improve coding capability. It does not fully define a separate mid-training schedule or give the size and mixture of that agentic corpus.

These priorities explain the intended coverage. They do not tell us exact proportions, source-level deduplication rates, contamination controls, or how every quality score is computed.

Packing and attention boundaries

Training examples are often packed together to use fixed-length sequences efficiently. If short documents are placed into a large sequence, padding can be reduced.

V4 uses sample-level attention masking. A token in one packed sample does not automatically gain access to a neighboring unrelated sample simply because both share a physical training sequence.

This matters for compressed attention. Compression blocks must respect sample boundaries. A summary should not pool the end of one sample with the start of another as if they were continuous text.

Trailing fragments shorter than the compression stride do not produce a completed compressed entry. Context-parallel communication must handle that irregularity.

The model can still learn from these trailing tokens through the ordinary token objectives and local representations. Discarding an incomplete compressed block does not mean dropping every such token from the language-model loss.

Tokenization and auxiliary formats

The paper keeps the vocabulary size at a rounded 128K and adds a few special tokens for context construction. It retains token-splitting and fill-in-the-middle strategies from the earlier model.

Fill-in-the-middle trains the model on missing text between known prefix and suffix material. This is useful for code editing and other completion tasks where the unknown span is not simply the final continuation.

The exact tokenizer vocabulary and formatting rules matter for reproduction. The released Pro configuration uses 129,280 entries; Chapter 22 records this as an implementation value supplementing the rounded paper description.

The reported optimization schedule

The batch grows from a smaller initial value to its maximum and stays there for most of training. The learning rate rises linearly, remains at its peak for much of the run, then decays with a cosine schedule.

The MTP weight drops when learning-rate decay starts.

Multiplying total tokens by the inverse of the maximum batch would give only a rough lower-bound-style step estimate because early batches are smaller. It would not recover the actual number or timing of steps.

Gradual context extension

Training starts at 4K sequence length and extends to 16K, 64K, and 1M.

Long sequences are expensive and require stable positional behavior, suitable data, and distributed memory management. Gradual extension lets much of training happen at cheaper lengths while introducing longer dependencies later.

The source does not specify the full token allocation for each length. It is therefore incorrect to say that every one of the 32T or 33T tokens is processed in a million-token example.

“Native million-token support” means the model's reported trained architecture and context-extension process support that window. It does not mean all training occurred at the final maximum length.

Introducing sparse attention

Flash starts with dense attention for the first 1T tokens. Sparsity is introduced at 64K length, with a short indexer warmup before most of the sparse training.

Pro uses a longer initial dense stage but the report does not give its exact length. It follows the same general two-stage approach to introducing sparsity.

Dense warmup reduces the initial dependence on a selector that has not yet learned useful rankings. The later warmup prepares the indexer before it becomes central to the attention path.

This motivation is plausible, but the V4 report does not show enough ablations to measure precisely how much each warmup contributes.

The word dense here concerns attention selection. The feed-forward modules remain MoE.

What caused loss spikes?

The authors link training spikes empirically to outliers in MoE layers and observe that routing appears to worsen them.

A rollback can return to an earlier checkpoint, but the same dynamic can recur. The paper therefore introduces a routing intervention and a direct activation bound.

The authors explicitly say the theoretical mechanisms remain insufficiently understood. This is useful honesty in the source: the proposed explanation is based on observation and successful mitigation, not a completed theory.

Anticipatory routing

At step $t$, feature computation uses the current model parameters $\theta_t$, while the routing indices come from historical parameters $\theta_{t-\Delta t}$.

To avoid loading two versions of the model during the same step, the system fetches step $t$'s data earlier. At step $t-\Delta t$, it runs a forward pass to compute and cache the routing indices that step $t$ will later use.

The intervention therefore decouples feature updates from routing decisions. It is not simply a moving average of router weights, nor a rule that predicts the next token's expert.

The paper describes historical network parameters for routing-index precomputation. A simplified explanation that only delays the final router's logits misses the dependence of routing on earlier hidden states.

Why delayed routing may help

Routing changes which expert weights receive a token and its gradients. If small changes in features or routing scores repeatedly redirect difficult tokens, the feedback can become abrupt.

Using earlier indices can reduce synchronous coupling between the current feature update and the current selection. This is a possible explanation for the observed stabilization.

It also introduces staleness. A rapidly changing network may receive routes that no longer fit its current features well. The fact that the authors use the method temporarily is relevant to that trade-off.

The source does not provide $\Delta t$, a full detector specification, or a controlled mathematical proof of the mechanism.

Automatic use after a spike

The system detects a loss spike, performs a short rollback, activates anticipatory routing for a period, then returns to normal training.

While active, the additional forward pass adds about 20% wall-clock overhead after optimization. The authors report negligible overall added overhead because the intervention is used dynamically rather than throughout the entire run.

These two claims refer to different denominators. A 20% increase during a small part of training can produce a much smaller increase over the full run.

The method also relies on deterministic infrastructure to investigate spikes and make comparisons. Without repeatable execution, separating a numerical issue from random variation is harder.

SwiGLU clamping

The expert's SwiGLU product can contain large values. V4 clamps the linear branch to $[-10,10]$ and caps the gate branch from above at 10.

The distinction between branches matters. The paper does not say to clamp both branches symmetrically to the same range.

Bounding the values before their multiplication limits extreme intermediate activations. It changes the function in the tails and can alter gradient flow where the clamp is active.

The authors report that this suppresses outliers and improves stability without harming performance in their runs. That finding should not be turned into a universal claim that clamping is always harmless for every architecture and threshold.

What the base-model results show

The paper evaluates base models before post-training under one internal framework with consistent settings. Flash beats V3.2 on many benchmarks despite lower total and active parameter counts. Pro improves further on most measures.

There are exceptions: V3.2 leads on BigCodeBench, and Flash leads Pro on some multilingual or Chinese math entries. Chapter 18 gives the numbers.

The broad improvement supports the complete pre-training recipe. It does not isolate whether the gain came from more data, better data, the architecture, Muon, or their interaction.

What a reproduction would still need

A serious reproduction would need the full data mixture, sample processing, context-stage budgets, dense-stage length for Pro, indexer training loss, routing intervention settings, precision rules, and distributed configuration.

The report provides a design and many practical lessons. It does not provide every operational input necessary to reproduce its weights or exact learning curve.

Source: local main.tex, Pre-Training, lines 904–1048. Interpretations of delayed routing and gradual context extension are identified as explanations, not additional measurements.

Continue with kernels and distributed training or return to the contents.

12. Kernels and Distributed Training

V4's architecture introduces irregular expert workloads, several attention representations, wider residual state, and a matrix-based optimizer. Efficient equations alone do not make these operations fast on a cluster.

The paper's systems work aims to keep computation, communication, and memory access busy together. It also makes results reproducible enough to debug training and align rollout with learning.

Expert parallelism

Expert parallelism distributes experts across devices. A token must reach the devices holding its selected experts, and their outputs must return to the token's originating position.

An MoE layer has four main stages:

1. Dispatch: transfer token activations to expert owners.
2. Linear-1: compute the expert's gate and up projections.
3. Linear-2: compute the down projection after activation.
4. Combine: return and combine the expert outputs.

Dispatch and combine are communication-heavy. The two linear stages are computation-heavy.

If the system runs each stage as a large separate phase, devices spend time waiting. Communication cannot hide behind computation when computation has not yet started.

Scheduling experts in waves

V4 divides experts into waves containing a small portion of the expert set. As soon as a wave's required communication completes, its computation can begin.

In a steady pipeline, one wave computes while the next wave receives tokens and a completed wave returns results.

time →
wave A: dispatch → compute → combine
wave B:      dispatch → compute → combine
wave C:           dispatch → compute → combine

This sketch describes overlap; it is not a measured timing chart.

The fused kernel coordinates these stages at a finer scale than a design that only overlaps dispatch with the first linear stage and combine with the second.

The paper names the open-source CUDA implementation MegaMoE and places it within DeepGEMM. It reports validation on NVIDIA GPUs and Huawei Ascend NPUs.

Reported speedups and their scope

Against strong non-fused baselines, the paper reports 1.50–1.73 times speedup for general inference workloads and up to 1.96 times for latency-sensitive workloads such as RL rollout.

These are MoE implementation results, not whole-model end-to-end speedups. Overall speedup depends on the fraction of elapsed time spent in MoE and on other bottlenecks.

If half of a task's runtime were accelerated by a factor of two while the rest stayed fixed, the total improvement would be

This derived example is Amdahl's law. It explains why local kernel gains should not be multiplied directly into an advertised model throughput.

Wave scheduling is especially relevant to long-tail small batches. An RL workload can contain requests that finish at different times, leaving a small set of continuing sequences with uneven expert traffic.

A compute-to-network balance

The report expresses a condition for hiding communication:

where $C$ is peak compute throughput and $B$ is interconnect bandwidth. The right side is arithmetic work per byte transferred.

For one token–expert pair in Pro, the main expert matrices use about $6df$ FLOPs. Dispatch uses FP8 activations and combine uses BF16 outputs, totaling about $3d$ bytes in this simplified accounting.

Their ratio is

since Pro's expert intermediate width is 3072.

The hidden width cancels. This is a useful dimensional check: the balance depends on expert intermediate width under these assumptions.

Once bandwidth is sufficient to hide the communication, increasing it alone gives diminishing benefit. Real systems also face latency, congestion, topology, and uneven workloads, so the ratio is a design guide rather than a sufficient guarantee in every case.

Other hardware observations

The fused workload activates compute, memory, and network at once. The authors observe that power throttling can then become a limiting factor.

Dispatch uses a pull-based approach: each GPU reads remote activations. The report links this choice to the notification latency of fine-grained push communication and suggests lower-latency cross-device signaling as a hardware improvement.

The authors also propose simpler element-wise activations without expensive exponential or division operations. This is a proposal for future designs. It does not mean V4 has already replaced SwiGLU with that proposed function.

Why use TileLang?

The architecture could create hundreds of small Torch operations. Each can add launch overhead and write intermediate tensors to memory.

TileLang allows the team to express and fuse custom kernels while retaining a higher-level development interface. The paper emphasizes three features: generated host launchers, stronger integer analysis, and numerical control.

The benefit is partly development speed. A model with several specialized attention paths needs many kernels, and a lower cost of experimentation makes architectural iteration more practical.

Generated host code

Even a fast device kernel can be slowed by Python-side shape, stride, and type checks. For a tiny operation, tens of microseconds of validation can dominate.

TileLang's Host Codegen creates a lightweight launcher together with the device kernel. It embeds the necessary contracts and uses TVM-FFI for argument handling and tensor interoperability.

The authors report CPU validation overhead dropping from tens or hundreds of microseconds to less than one microsecond per invocation.

This concerns host validation overhead. It is not a claim that every complete kernel invocation takes less than one microsecond.

Integer analysis with Z3

Compilers must reason about indices, bounds, layouts, and memory hazards. For a complicated tensor expression, proving that a transformation is safe can be difficult.

The team integrates Z3 using quantifier-free nonlinear integer arithmetic. This can help with vectorization, barrier placement, and code simplification.

The report says the added compile-time cost is restricted to a few seconds under reasonable limits. The analysis runs during compilation, not as a solver call for every generated token.

Stronger analysis can enable better kernels, but it should not be confused with a proof that the entire model implementation is correct.

Numerical control

TileLang disables fast-math optimizations by default and exposes approximate operations as explicit opt-ins. It also offers IEEE-oriented intrinsics and rounding controls.

The team aligns algebraic simplification and lowering with CUDA toolchains and allows layout annotations to fix accumulation behavior.

Floating-point addition is not associative. Changing reduction order can change low bits, which can change later rankings or sampled outcomes.

Numerical agreement is therefore a systems goal as well as a debugging convenience.

Batch invariance

Batch invariance means a token's output is bitwise unchanged by its placement within a batch.

This is stronger than saying model weights are unchanged. Kernels can choose different reduction strategies for different batch shapes, producing different numerical results.

For attention decode, V4 uses a dual-kernel strategy. One kernel computes an entire sequence's attention within one streaming multiprocessor for full waves. A second uses multiple SMs for the last partially filled wave while matching the first kernel's accumulation order.

The second kernel uses distributed shared memory within thread-block clusters for fast exchange. The purpose is to improve final-wave utilization without giving up bitwise agreement.

For matrix multiplication, the team uses DeepGEMM instead of cuBLAS and avoids ordinary split-k in most cases. It optimizes small batches to compensate for the lost parallelism.

Deterministic backward passes

Sparse attention backward can have many queries updating the gradient of the same KV entry. Atomic additions can accumulate in nondeterministic order.

V4 allocates separate accumulation buffers per SM and then sums them in a deterministic global order.

For MoE backward, token order preprocessing and cross-rank buffer isolation stabilize sending positions and accumulation.

mHC has a projection output of only 24 coefficients. For small batches, split-k can still be necessary. The implementation writes split results separately and reduces them in a later deterministic kernel.

Deterministic execution does not make sampled text deterministic by itself. Sampling also requires controlled randomness, and any external tools can introduce their own variation.

Sharding whole matrices for Muon

Muon needs the complete logical matrix for an update. V4 therefore uses a hybrid ZeRO assignment.

Dense parameters are assigned as whole matrices to a limited set of ZeRO ranks using a knapsack algorithm. Buckets are padded to the largest size so reduce-scatter works efficiently. The reported padding overhead is usually below 10%.

When data parallelism exceeds the ZeRO limit, extra groups redundantly compute Muon updates. This trades some compute for smaller bucket memory.

Expert matrices are numerous enough to distribute without splitting each logical matrix. Identically shaped consecutive parameters are batched for Newton–Schulz iterations.

Low-precision gradient synchronization

The authors find BF16 matrix multiplication stable for Newton–Schulz iterations. They also synchronize MoE gradients in BF16 with stochastic rounding, halving communication compared with FP32 payloads.

To avoid repeated low-precision additions, they replace a conventional tree or ring reduction with all-to-all exchange followed by a local FP32 sum.

The precision of transfer and the precision of accumulation are different choices. Sending BF16 does not require accumulating the sum in BF16.

Stochastic rounding reduces systematic rounding bias in an appropriate setup; it does not make every quantized value exact.

Context parallelism for compressed attention

Context parallelism partitions the sequence across devices. Compression introduces two complications: sample boundaries create variable numbers of completed blocks, and a block can cross a rank boundary.

V4 uses two communication stages. First, a rank sends its last $m$ uncompressed entries to its neighbor. The neighbor combines boundary information with local tokens and creates a padded fixed-length compressed output.

Second, the ranks all-gather compressed entries. A fused select-and-pad operation organizes valid entries and puts padding at the tail.

The approach reduces the large global exchange to compressed representations after the local boundary problem is handled.

Sparsity then determines which gathered entries each query may read. Correct sample and causal masks remain necessary.

Tensor-level activation checkpointing

Training needs forward activations for backward gradients. Storing all of them consumes memory; recomputing all of them consumes time.

Conventional checkpointing often chooses whole modules. V4 instead annotates tensors and traces the graph with TorchFX. The framework finds the minimal recomputation subgraph for an annotated tensor and inserts it before the relevant backward computation.

It can track storage pointers and avoid duplicate recomputation for views sharing storage. Recomputed storage is reused without a GPU copy.

The paper reports no extra training overhead compared with manual implementation of the same recomputation policy. That does not mean recomputation itself is free.

This mechanism supports mHC's expanded state and the long-context memory demands while retaining automatic differentiation.

Source: local main.tex, General Infrastructures, lines 595–821. Numerical examples and the Amdahl calculation are derived explanations.

Continue with serving and prefix reuse or return to the contents.

13. Serving and Prefix Reuse

An inference server must preserve enough state to continue each request. In V4, that state is heterogeneous: different layers compress at different rates, CSA has an indexer cache, and recent tokens remain uncompressed.

Treating every layer as a conventional ever-growing KV sequence wastes memory or violates the assumptions of optimized kernels. The paper separates growing history from bounded sequence state.

The kinds of cached state

The exact buffers can contain projected values, compression logits, and other needed state. The paper gives the management principle rather than a full allocation manifest.

Why a conventional paged cache is not enough

PagedAttention-style systems allocate cache blocks and map logical sequence positions to them. A straightforward implementation often assumes a fixed token count per cache block for all relevant layers.

V4 breaks this uniformity. A block covering a fixed count of original tokens creates many CSA entries and far fewer HCA entries. The local window also evicts old positions, while compressed history remains available.

Different layer policies and kernel alignment constraints make a single unmodified paging rule awkward.

The paper responds with a growing classical cache for completed compressed entries and a fixed-size state cache for the window and unfinished compression.

Bounded state per request

The state cache gives each active request a fixed-size block. Part of it stores recent local KV entries; another part stores the unfinished compression state.

This state depends on the request's current position. When a new token arrives, the window advances and the compression state changes. When a block completes, a summary moves into the growing historical cache.

The bounded pool limits how many simultaneous requests can be assigned state, even when their completed compressed histories are small.

This is a familiar distinction in streaming systems: an append-only history can coexist with a small mutable state that describes the current frontier.

Aligning original-token blocks

For compression rates $m$ and $m'$, a natural allocation unit covers a multiple of

With 4 and 128, the least common multiple is 128 original tokens.

One such block yields 32 CSA entries and one HCA entry. A larger allocation unit could cover several of these groups.

This gives a shared original-token boundary across layers. It does not make the per-layer byte sizes equal.

The paper says sparse-attention kernels are co-designed with cache layout to accommodate different numbers of entries per block. Padding for cache-line alignment can help performance.

Logical compression units, physical allocation units, and kernel tiles are related but distinct. Mixing them leads to incorrect memory or mask calculations.

Shared-prefix reuse

Many requests start with the same material: system instructions, a large repository snapshot, or a document collection. Recomputing all of it for every request is wasteful.

V4 stores KV state on disk so a request hitting a known prefix can reuse it. The compressed history is small enough to make storing more historical state attractive.

A prefix hit should be tied to the exact relevant inputs and model settings. Changed text, tokenizer behavior, weights, or positional treatment can invalidate the cached representation. The paper does not provide a complete cache-key specification.

Caching is also subject to capacity and I/O limits. A hit that requires slow disk loading may still save large prefill work, but it is not a zero-cost operation.

Completed summaries and incomplete tails

The server can load completed CSA and HCA entries directly. It cannot load an unfinished compressed block from the completed-summary store if the uncompressed tail was not saved.

Suppose a shared prefix has 1,000 tokens. CSA has 250 complete stride-four blocks. HCA has seven complete stride-128 blocks and a 104-token tail.

To continue HCA compression correctly, the server must restore the unfinished-tail state. The local sliding window also needs recent uncompressed entries.

This example shows why “the compressed cache is available” is not enough to begin decode. The continuation needs frontier state as well.

The much larger local-history store

At runtime, a sliding window needs only recent entries. If the server saves every token's local KV entries on disk for future prefix reconstruction, it stores an uncompressed sequence in every layer.

The report says this full SWA disk data is about eight times larger than the compressed CSA/HCA entries.

There is no contradiction with a small runtime window. Runtime resident size and total historical disk size are different quantities.

Most prefix hits use only the final window segment. Saving every historical local entry creates many writes for data rarely read again.

Strategy 1: full SWA caching

The server stores all local KV entries for every token.

On a prefix hit, it loads the last $n_{\mathrm{win}}$ positions needed by the current window. This avoids recomputing those local entries.

The cost is disk traffic and storage. The authors highlight an unbalanced write-heavy access pattern on SSD-based systems: a large volume is written, but only a small final segment is often read.

“Zero recomputation” in this strategy concerns the saved SWA state. Incomplete compressed tails still need the treatment described for the compressed branch.

Strategy 2: periodic checkpoints

Every $p$ original tokens, save the last $n_{\mathrm{win}}$ local entries. When a request hits a prefix, load the nearest suitable checkpoint and recompute the remaining tail.

Smaller $p$ means more storage and less replay. Larger $p$ means fewer checkpoints and more replay.

For a simple uniform-prefix-length model, the distance to the last checkpoint would average roughly $p/2$. Real workloads need not follow that distribution. Popular prefixes can cluster near specific lengths.

Choosing $p$ is therefore a workload decision. It should use measured prefix lengths, I/O throughput, recomputation speed, and reuse frequency.

Strategy 3: zero SWA caching

The server saves compressed history but no historical SWA entries.

To rebuild the final local window of an $L$-layer model, the paper says recomputing the last

original tokens is sufficient when the compressed history is already available.

The reason is a dependency cone. To compute the needed local state at one layer, one needs a preceding window of local state from the previous layer. Moving down a layer expands the required original-token range by at most another window.

With a 128-token window, the bound is 7,808 tokens for 61 layers and 5,504 for 43 layers.

These are sufficient recomputation bounds from the paper's argument, not proof of the exact minimum for every request.

Why the cone argument needs cached global memory

A Transformer layer also reads compressed history. Without cached compressed entries, rebuilding a tail could require rebuilding much older information.

The bounded replay argument assumes those global compressed representations are already available. It also assumes a causally correct replay schedule and restoration of any needed compression-tail state.

Feed-forward experts act within a token position. mHC mixes streams within that position. They do not independently extend the sequence dependency cone in the way another local attention layer does.

The argument is useful for storage planning, but an actual server must still validate boundary cases and cache reconstruction numerically.

Comparing the three strategies

The paper says it selects strategies by deployment scenario. It does not declare one globally optimal policy.

Frequent reuse of a stable long prefix can justify more storage. Rarely reused prefixes can make extensive caching wasteful.

What to measure in a server

A cache design should be evaluated by hit rate, bytes read and written, reconstruction latency, memory fragmentation, and concurrent-request limits.

Quality and numerical consistency matter too. A resumed request should behave like a request that prefills the same prefix under the intended precision and deterministic settings.

Incorrect restoration may first appear as a subtle change in attention outputs rather than an obvious crash. Block-boundary tests and exact-prefix continuation checks are therefore meaningful correctness checks.

What the paper does not quantify

The source does not give a complete distribution of prefix hits, disk bandwidth measurements, checkpoint intervals, or total serving latency for every cache strategy.

It explains a design space and the dependencies that make the strategies possible. An operator still needs workload-specific measurements to choose a policy.

Source: local main.tex, Inference Framework, lines 823–902. The 1,000-token example, layer-specific replay counts, and checkpoint-distance example are derived.

Continue with specialists and rewards or return to the contents.

14. Specialists, Rewards, and Reasoning Modes

Pre-training teaches a model to predict language. Post-training teaches it to answer requests, use tools, and solve tasks in the forms users need.

V4 first trains specialists separately, then consolidates their behavior into a single model. Specialists are full model policies trained for domains such as mathematics, coding, agent tasks, and instruction following.

Why train specialists separately?

Different tasks have different success criteria. A formal proof can be checked by a verifier. A program can be tested. A writing request may need a judgment about tone, completeness, and adherence to constraints.

Training these together creates competing signals and different operational needs. Separate specialists let each domain use its own data, reward, and reasoning budget.

The final user-facing model still needs many skills. V4 uses on-policy distillation to transfer the specialists' behavior into one parameter set.

This design gives up the simplicity of one mixed RL run in exchange for easier domain-specific development and an additional consolidation stage.

Supervised fine-tuning

A specialist first learns from high-quality domain examples. Given a prompt $x$ and target response $y$, a standard SFT loss is

This teaches response structure, useful habits, and initial task competence.

SFT is limited by the provided examples. It rewards matching a target continuation even when another valid response would solve the task. It also conditions on the target's earlier tokens, which can differ from the histories the model creates when generating independently.

RL then lets the model explore responses and receive outcome-based feedback.

GRPO in broad terms

The paper uses Group Relative Policy Optimization, GRPO, with settings close to prior DeepSeek work.

A typical GRPO setup samples a group of responses for a prompt, scores them, and converts their relative rewards into advantage estimates. A common illustrative form is

The policy is encouraged to increase probability for better-than-group responses and reduce it for worse ones, subject to the method's clipping and regularization.

This equation explains the group-relative idea. The V4 report does not restate every precise GRPO normalization, clipping, or KL setting, so it should not be treated as a recovered implementation.

Relative advantages can reduce the need for a separately trained value function. They also depend on group composition. If every sample gets the same reward, there may be little useful within-group signal.

Verifiable tasks

For coding, tests can provide a success signal. For formal mathematics, a strict verifier can establish whether a proposed proof is accepted. For some short-answer tasks, an answer checker can compare an extracted result.

These rewards can be less subjective than preference judgments. They still require careful task design. Weak tests can reward incomplete solutions. A parser can misread a correct answer. A verifier can enforce a formal target that differs from the intended informal problem if the formalization is wrong.

The quality of the reward interface is part of the training method. A model optimizes the signal it receives, not an unstated human intention.

Hard-to-verify tasks

A professional report or creative response usually has no single exact answer. Human preferences often involve several dimensions.

V4 uses rubric-guided RL data and a Generative Reward Model, GRM, rather than conventional scalar reward models for these tasks.

A generative evaluator can read a prompt, response, and rubric, reason about them, and produce an evaluation. A scalar model instead directly predicts a numerical reward through a specialized output.

The generative form allows the evaluator to use the model's language and reasoning abilities. Its explanation can make an evaluation easier to inspect. The explanation does not make the judgment automatically correct.

The actor also serves as evaluator

The source says the actor network natively functions as the GRM, and that RL directly optimizes the GRM as well.

This joins generating and judging within the model's capabilities. Learning to evaluate constraints can support generation, and generation knowledge can support evaluation.

The report claims robust scoring from a small diverse set of human annotations. It does not fully disclose the scoring prompt, annotation scale, judge-training objective, or controls for correlated evaluator errors.

A useful open question is whether the evaluator has a bias toward the actor's own style. Another is how its scores remain calibrated as the actor changes.

The book can explain the proposed arrangement, but cannot fill these missing methods with an assumed implementation.

Reasoning effort is trained behavior

Both Flash and Pro support three modes:

The specialists are trained with different length penalties and context windows. The resulting policies learn different preferences about how much reasoning to generate.

Max also uses a system instruction emphasizing thorough analysis and checking. The prompt is part of the mode, but the paper does not claim the prompt alone creates the capability without training.

Non-think does not imply that the neural computation contains no inference. It means the model does not produce the same explicit reasoning-token path.

Length penalties and quality

Without a cost for length, a rewarded model can generate very long responses. Longer reasoning can help solve hard tasks, but it can also repeat itself or explore unproductive branches.

A length penalty trades task reward against generation cost. Different penalties can teach different effort levels.

The report does not give all penalty formulas or coefficients. We can identify the control mechanism but cannot calculate an exact optimal response length from the source.

Reasoning length should therefore be evaluated alongside accuracy and elapsed time. A longer trace is not itself evidence of a better solution.

Training windows and evaluation windows

For knowledge and reasoning evaluation, the paper uses 8K, 128K, and 384K windows for Non-think, High, and Max. These settings can constrain the total available sequence, not merely the length of the final answer.

Agent evaluation uses a different maximum context of 512K. The model architecture supports a larger maximum.

Three quantities should remain separate: architectural context capacity, evaluation context budget, and actual generated token count.

If one model uses more test-time compute, a raw accuracy comparison answers “which tested system scored higher,” not necessarily “which model is more efficient per token or per second.”

Why specialists still need consolidation

A mathematics specialist may excel at proofs but be less suited to a short writing task. A tool specialist may be better at long action sequences but need other teachers for factual or stylistic tasks.

Serving every specialist behind a task router creates its own system and user-experience problems. Weight merging can interfere with learned behavior. Mixed RL can make skill balancing difficult.

V4's final student learns from more than ten teachers at the distribution level. That is the focus of the next chapter.

What this stage establishes

The method offers a clear development structure: teach initial behavior, optimize domain outcomes, train several effort settings, then consolidate.

The paper does not provide isolated specialist scores for every stage, a full reward-data inventory, or a complete analysis of evaluator reliability. Final benchmark improvements support the overall post-training result, while leaving those mechanisms partly open.

Source: local main.tex, Post-Training Pipeline and Specialist Training, lines 1053–1124; reasoning formats, lines 1063–1110; OPD overview, lines 1218–1237. SFT and group-relative equations are background explanations, not extra disclosed V4 settings.

Continue with on-policy distillation or return to the contents.

15. On-Policy Distillation

After domain specialists are trained, V4 consolidates their skills into one model with multi-teacher on-policy distillation, OPD. This replaces the previous mixed-RL stage.

The student generates trajectories. Relevant teachers evaluate the distributions at states along those trajectories. The student then learns to align its probabilities with theirs.

Why student-generated histories matter

Imagine a teacher solving a programming problem. It chooses a good design, writes a correct helper function, and reaches a clean final answer.

A student may choose a weaker design or write a faulty helper first. The states it then visits are different. Training only on the teacher's clean trajectory does not directly teach the student how the teacher would behave in those student-created states.

On-policy distillation asks the teacher about the student's own histories. This reduces the mismatch between training contexts and the contexts produced during student generation.

It does not guarantee recovery from every error. Some histories can already be unsalvageable, and the training prompts and sampling policy limit which states are visited.

The reported objective

The paper writes

where $\pi_\theta$ is the student and $\pi_{E_i}$ is a teacher.

The student distribution appears first. This is reverse KL relative to the common teacher-to-student distillation direction.

At a fixed history $h$, let $p(v)$ be the student's next-token probability and $q(v)$ the teacher's. The full-vocabulary loss is

This compares all vocabulary probabilities at that position, not only the token sampled by the student.

What reverse KL penalizes

Suppose the student puts substantial probability on a token the teacher considers very unlikely. The ratio $p(v)/q(v)$ becomes large, increasing reverse KL.

If the teacher supports several alternatives but the student concentrates on one valid alternative, reverse KL can tolerate that concentration more readily than forward KL.

Forward KL,

heavily penalizes the student for neglecting alternatives on which the teacher puts mass.

The terms “mode seeking” and “mode covering” are useful tendencies, not universal guarantees about every neural optimization run.

Reverse KL cannot fix a teacher's error. If the teacher assigns high probability to an incorrect continuation, aligning with it can transfer that mistake.

A small distribution example

Let the teacher assign probabilities $(0.49,0.49,0.02)$ to three possible tokens. Two are plausible; the third is poor.

Compare students

and

Student A largely chooses one teacher-supported option. Student B covers both good options but puts much more mass on the poor option.

For these particular values, the reverse KL losses are approximately 0.388 for A and 0.084 for B, using natural logarithms. B is closer overall despite giving more mass to the poor option. A's strong concentration is costly in this example.

The example shows why both the KL direction and the actual probabilities matter. A tendency toward mode selection does not mean concentration on one plausible token is always rewarded.

It is important not to turn a slogan about KL direction into a claim that one objective always produces higher benchmark accuracy.

Token-level and sequence-level KL

For autoregressive sequence distributions $P$ and $Q$, the chain rule gives

with compatible treatment of ending and sequence length.

This explains why histories sampled from the student are relevant. At those histories, a full-vocabulary next-token KL gives a conditional term in the sequence comparison.

Training introduces another question: how gradients handle the dependence of the sampled histories on the student. Differentiating only the conditional distributions at fixed sampled histories is not identical to automatically differentiating every part of the full expectation.

The report does not fully state all estimator and stop-gradient choices for this dependence. We should not infer an exact sequence-level gradient implementation from the compact objective alone.

The sampled-token alternative

A sampled token $y_t\sim p$ gives the estimate

whose expectation is the fixed-history reverse KL.

Prior methods can reuse an RL framework by turning a stopped-gradient teacher-to-student log ratio into a per-token advantage.

The V4 report says this sampled-token approach has high gradient-estimation variance and can cause instability. It instead uses full-vocabulary logit distillation.

The full sum removes the sampling approximation over vocabulary items at a fixed history. It does not remove randomness in prompts, student trajectories, batching, or optimizer updates.

What “logits-level” supervision adds

A teacher-written target token says which token occurred. A full probability distribution says how the teacher ranks alternatives and how confident it is.

For a naming choice in code, several tokens may be acceptable. For an exact mathematical answer, the teacher may strongly prefer one continuation. These distributions provide a richer signal than a single target.

Matching logits requires compatible tokenization and vocabulary indexing, or an explicit mapping. The report describes full-vocabulary matching within its own model family rather than a general procedure for arbitrary incompatible tokenizers.

The teacher's distribution also depends on temperature and formatting. Those choices should be specified in a reproduction; the V4 source does not give every such setting in the OPD subsection.

Which teacher teaches which task?

The source describes selective learning from the specialist relevant to the task: mathematics for math, coding for programming, and so on.

The formal weighted sum expresses the overall multi-teacher objective. It should not be read as a requirement that every teacher runs on every token of every trajectory.

Task mixture weights determine which capabilities receive more training. Within-domain teacher selection can also matter.

More than ten teachers are used. Their number alone does not establish diversity, quality, or complete coverage of user requests.

If all teachers were combined at one fixed state

For nonnegative weights summing to one, minimizing

at a fixed state is equivalent, up to a constant, to matching a normalized geometric mixture:

This follows by collecting the terms involving $\log q_i(v)$ in the KL sum.

The geometric mixture favors tokens supported jointly by the teachers. It differs from an arithmetic average of probabilities.

This is a mathematical interpretation of the displayed objective under the stated fixed-state assumptions. It is not a claim that V4 literally constructs this mixture in its implementation. The report instead emphasizes relevant-teacher learning across tasks.

Why full logits are expensive

Let a trajectory contain $T$ positions and the vocabulary contain $|V|$ tokens. Materialized logits have $T|V|$ entries per teacher.

With a rounded 128K vocabulary, one million positions produce 128 billion entries. At two bytes per entry, that is 256 GB for one teacher's logits, before other state.

This is a scale illustration, not a claim that every training sample has one million positions or that all logits are kept at once.

Several teachers make naive storage still less attractive.

Cache hidden states instead

V4 stores the teachers' last-layer hidden states in a centralized buffer. It reconstructs logits later by applying the appropriate prediction head.

For Pro hidden width 7168, one million two-byte hidden vectors would occupy about 14.3 GB in a simplified calculation. That is still large, but far smaller than a full 128K vocabulary vector at each position.

The calculation assumes BF16-like two-byte storage for illustration; the paper does not specify every buffer dtype in this passage.

Recomputing the final head is relatively cheap compared with recomputing the full teacher, and it avoids persistent full-logit storage.

Teacher scheduling and sharding

Teacher weights are offloaded to centralized distributed storage and loaded on demand using ZeRO-like parameter sharding.

Samples are ordered by teacher index. This lets the runtime load one prediction head at a time, using it for the relevant samples before switching.

Weights and hidden states move asynchronously so I/O can overlap computation. A specialized TileLang kernel computes exact full-vocabulary KL terms without large unnecessary intermediate allocations.

“Exact” here distinguishes the full vocabulary sum from a sampled-token estimate. Floating-point arithmetic and finite precision still apply.

What OPD does not establish by itself

Distillation can transfer teacher behavior into one model, but capacity and conflicting tasks can still create trade-offs. A student cannot be assumed to exceed every teacher on every domain.

The final scores support the effectiveness of the combined pipeline. They do not quantify skill retention for all individual specialists because those teacher comparisons are not fully reported.

A strong assessment would measure each teacher, the student before consolidation, the student after consolidation, and matched mixed-RL alternatives. It would also compare training cost and task-mixture sensitivity.

Source: local main.tex, On-Policy Distillation, lines 1218–1237; teacher scheduling, lines 1266–1276. KL derivations and storage examples are explanatory calculations. The source does not fully disclose trajectory-gradient handling.

Continue with quantization-aware training or return to the contents.

16. FP4 Quantization-Aware Training

V4 uses low precision to reduce weight memory, cache traffic, and long-context selection cost. The paper introduces FP4 quantization-aware training during post-training so the model can adapt to the representation used for rollout and serving.

Low precision is applied selectively. The report does not describe the entire model, every activation, and every optimizer state as FP4.

The two main FP4 targets

The paper applies MXFP4 to:

1. MoE expert weights, a major part of model storage.
2. The CSA indexer's query–key path, where QK activations are cached, loaded, and multiplied in FP4.

It also quantizes index scores from FP32 to BF16 for top-k selection.

Main KV storage uses FP8 for non-rotary dimensions and BF16 for rotary dimensions. Optimizer master weights remain FP32 in the described QAT path.

These choices address different costs. Expert weight quantization reduces memory traffic and model footprint. Indexer quantization reduces the cost of scanning a long compressed history. Lower-precision scores reduce selector traffic and processing.

Quantization is a representation change

A quantizer maps many real values to a smaller set of representable values. A typical block-scaled representation has the form

where $q$ is a low-precision value and $s$ is a shared block scale.

The scale lets a small numeric format represent values appropriate to a local range. It also adds metadata and affects which values are rounded.

FP4 E2M1 has far fewer representable magnitudes than FP8 E4M3. Its exact format and block scale rules matter. Saying “four-bit weights” without the scale structure is not a complete numerical specification.

Why use fine-grained scales?

A large block can contain a few large values and many small values. A scale chosen for the large values can cause small ones to round poorly.

Smaller scale blocks adapt to local ranges. They improve representation at the cost of more scale metadata and more handling in kernels.

The described FP4 blocks are $1\times32$ tiles. The existing FP8 quantization blocks are $128\times128$ tiles.

The difference in granularity is important to the claimed lossless FP4-to-FP8 conversion.

Post-training must adapt to the quantized model

If a model is trained entirely in a higher-precision representation and only quantized after training, its behavior can change.

QAT inserts quantization into the training path. The learned weights can adjust to the rounding and scaling constraints.

This is particularly relevant to RL and OPD. The student generates trajectories under a rollout model. If the learner uses materially different effective weights, the training and sampling policies can disagree.

The paper applies quantization-aware treatment to teacher and reference models as well. This helps align the models involved in the post-training comparison.

QAT can reduce degradation. It does not make the low-precision representation identical to the original master weights.

The FP32 to FP4 to FP8 path

During training, the optimizer keeps FP32 master weights. The forward path quantizes them to FP4, then converts the quantized representation to FP8 for computation.

FP32 master weights
  → FP4 quantization with fine-grained scales
  → FP8 representation of those quantized weights
  → existing FP8 training kernels

The purpose of the last conversion is to reuse the existing FP8 training framework.

The conversion does not remove the FP4 quantization error. It represents the already-quantized values in a wider format.

What “lossless dequantization” means

FP8 E4M3 has more exponent bits than FP4 E2M1. Under suitable scale relationships, the fine FP4 block scales can be absorbed into FP8's larger dynamic range.

The paper says the ratio between the maximum and minimum FP4 sub-block scales within an FP8 block must stay below a threshold. It reports that the current weights satisfy the condition.

Thus, the FP4-represented values can be moved into the FP8 computation format without further loss.

This is a conditional claim about the second conversion:

It is not a claim that

is lossless.

The report does not give the complete threshold and numerical proof in this subsection. A faithful implementation needs those details.

The straight-through estimator

Rounding is not smoothly differentiable. A quantized forward pass therefore needs a practical way to update the master weights.

V4 computes gradients using the effective FP8 weights from the forward path and propagates them directly to FP32 master weights. This is equivalent to a straight-through estimator, STE, across quantization.

A schematic update is

with the optimizer's actual update replacing the simple gradient step.

The estimator treats the effective quantization path as if an appropriate gradient can pass through it. It is not the exact derivative of discrete rounding.

It also avoids a separate re-quantization of transposed weights in the backward path described by the paper.

Native FP4 in rollout

Rollout and serving have no backward pass. They can use native FP4 weights directly rather than simulating them through FP8.

The paper says this reduces memory loading and occupancy while keeping sampling behavior aligned with deployment.

The exact agreement still depends on kernel arithmetic, scales, and batch invariance. The broader infrastructure is designed to support that consistency.

Native storage can help even if the hardware's peak arithmetic throughput does not increase. Fetching half as many scalar weight bits reduces traffic before accounting for scales.

BF16 index scores

The selector receives scalar scores for many candidate blocks. V4 converts these scores from FP32 to BF16 during QAT.

The authors report a twofold top-k-selector speedup and 99.7% recall of KV entries.

The recall figure concerns overlap with the reference-selected entries under the measured setup. It is not a 99.7% answer-accuracy claim.

Two scores close together can change order when rounded. QAT lets the model adapt to this selection behavior, but task-level consequences still require evaluation.

The source does not fully specify the dataset, tie rules, or aggregation used for the recall measurement in that passage.

Weight-memory estimates

If every parameter in a 1.6T model used four bits, the scalar payload would be

For 284B parameters, it would be 142 GB.

These are optimistic all-parameter payload illustrations, not actual V4 deployment footprints. Some weights use higher precision; scales, metadata, padding, and runtime buffers add memory.

They show why total parameters remain important even when only 49B or 13B are active per token.

Training needs far more than inference payload. Master weights, gradients, optimizer state, activations, and communication buffers must also fit, usually through sharding.

Precision and performance

The paper notes that current peak throughput for FP4-by-FP8 operations matches FP8-by-FP8 on the referenced hardware. It discusses a possible future one-third efficiency improvement.

This means there are at least two separate benefits to examine: memory traffic and arithmetic hardware support.

A deployment can benefit from lower weight bytes even without a doubled arithmetic peak. Conversely, a fast low-precision matrix unit can be underused if expert batches are too small or dispatch stalls.

The headline FLOP comparison uses equivalent FP8 accounting, which should not be mixed directly with raw operation counts or a universal bit-width speed multiplier.

What evidence would isolate QAT's effect?

A controlled evaluation would compare the same trained model in higher precision, post-training quantization, and QAT, using matched tasks and runtime settings.

For the indexer, it would separately measure ranking agreement, task accuracy, scan latency, and selector latency. For expert weights, it would measure memory, throughput, and behavioral changes.

The report gives the method and several local efficiency measurements. It does not provide every isolated quality comparison needed to attribute final scores to quantization choices.

Source: local main.tex, FP4 Quantization-Aware Training, lines 1246–1264; mixed KV storage, lines 533–545; future hardware discussion, lines 223–227. Weight-memory calculations are deliberately simplified illustrations.

Continue with agents and rollout or return to the contents.

17. Agents, Conversation State, and Rollout

A long-running agent needs more than a large context window. It needs reliable tool formatting, useful context retention, recoverable generation, and an execution environment that can survive interruptions.

V4 addresses each of these at a different level: model behavior, conversation management, inference infrastructure, and sandbox infrastructure.

Tool calls are model output

A model chooses tools by generating a structured representation that an application parses and executes.

V4 uses an XML-style DSML format with a dedicated marker. A simplified example is:

<|DSML|tool_calls>
<|DSML|invoke name="search">
<|DSML|parameter name="query" string="true">compiler error details</|DSML|parameter>
<|DSML|parameter name="limit" string="false">5</|DSML|parameter>
</|DSML|invoke>
</|DSML|tool_calls>

The tool name and parameter names must match the schemas provided by the application.

String values use the string flag. Other types, such as numbers, booleans, objects, and arrays, are passed in JSON form with the flag set false.

The paper reports fewer escaping failures and tool-call errors with this format. It does not provide a full per-format error-rate table in the source passage.

Syntax and task success are separate

A correctly parsed call can still choose the wrong tool, use the wrong arguments, or misinterpret the result.

The format addresses an interface failure: malformed output. Training and evaluation must also address decision failures.

For code agents, a long tool-output transcript can create another difficulty. The model must identify a small relevant error among many lines. The context window keeps the information available, but attention and task behavior determine whether it is used.

Reliable tool use therefore depends on both parser correctness and model competence.

Interleaved thinking

The model can generate reasoning text between observations and actions. The conversation manager decides which of that text remains in later context.

V3.2 retained reasoning across tool-result rounds but discarded earlier reasoning when a new user message arrived.

For V4 tool-calling scenarios, the report preserves reasoning across the whole conversation, including user-message boundaries. For ordinary conversation, it keeps the earlier policy of discarding past reasoning when a new user message arrives.

These are application context rules, not a guarantee that every API or third-party harness automatically uses the same policy.

Why retention can help

An agent may have already compared designs, rejected an approach, or linked an error to a particular file. If that state disappears, the agent may repeat work or lose the reason behind a decision.

Preserving it can reduce repeated reconstruction. It also consumes context and carries earlier mistakes forward.

A large window makes retention less costly, but does not remove the need for task-state management. Important decisions should still be reflected in artifacts and verified results where possible.

The paper's policy is a choice about balancing continuity and context size.

Simulated tool messages can change the path

Some frameworks place tool-like observations in user messages rather than using the intended tool-result structure.

The report notes that these frameworks may not trigger the tool-calling retention path. It recommends non-think models for such architectures.

This is an example of how a harness changes model behavior. Two applications can send similar visible text yet create different internal conversation structures.

An evaluation that changes message roles or strips reasoning history is testing a different operational setup from one that preserves the paper's intended path.

Quick Instruction

A chatbot often performs small auxiliary tasks before answering: decide whether search is needed, create queries, identify a domain, or generate a conversation title.

A separate small model would need its own prefill of the relevant conversation. V4 appends dedicated task tokens so the same model can reuse an existing KV cache.

The reported tokens include action, title, query, authority, domain, and URL-reading decisions.

Some tasks can run in parallel from the shared context. Conceptually, they branch from a common prefilled prefix; they need not be interpreted as one long sequential chain of appended instructions.

The model still performs additional computation for each task. The saving is avoiding repeated prefill and a separate auxiliary-model lifecycle.

Rollout is a training service

RL and OPD require generated trajectories. A rollout engine samples model tokens and, for agents, interacts with external environments.

Long responses make rollout expensive and vulnerable to interruption. A shared GPU cluster can preempt jobs for higher-priority work. Hardware can also fail.

If a request is interrupted, the training system must preserve the distribution it intended to sample from. Recovery is not only an operational matter; it can affect learning.

Why restarting from scratch creates length bias

Suppose an interruption process has approximate rate $\rho$ per unit time, and a response takes duration $T$. Its chance of finishing without interruption is

If interrupted attempts are discarded and fresh responses are sampled until one finishes, short responses are more likely to enter the completed dataset.

The observed duration distribution is then tilted approximately as

This is an illustrative model of the bias, not the paper's exact failure-rate measurement.

The consequence is clear: a training system can accidentally favor short outputs without changing the explicit reward.

Token-granular write-ahead logs

V4 logs every generated token immediately in a request-specific write-ahead log, WAL.

During preemption, it pauses the engine and saves KV caches for unfinished requests. Resumption continues from the saved tokens and state.

After a fatal hardware error, persisted tokens can be prefetched through the model again to rebuild the KV cache. The completed portion of the sampled trajectory is preserved.

The system avoids drawing a completely new response merely because the old one was interrupted.

If sampling randomness and batch-invariant deterministic execution are controlled, regeneration with the same random seed can also preserve the intended trajectory. The authors prefer WAL because replaying all previously generated decode steps wastes more work.

Token recovery and tool recovery differ

A token log can reconstruct model state. It cannot by itself reverse or recreate an external file edit, database change, or shell command.

An agent therefore also needs an execution-state log. Re-executing a completed command can have a different effect, especially when the command is not idempotent.

The model-generation service and the sandbox service address these different recovery layers.

Data handling for million-token training

Large rollouts contain both light metadata and heavy per-token fields.

V4 loads global metadata for shuffling and packing decisions. It loads heavy fields through a shared-memory data loader and releases them at mini-batch granularity after use.

This avoids redundant copies within a node and limits memory pressure. The number of resident mini-batches is adjusted to balance throughput against I/O overlap.

Packing long agent trajectories is also a correctness task: action and observation ordering must remain intact, and token-level fields must stay aligned with the corresponding sequence.

DeepSeek Elastic Compute

The paper's sandbox platform is DeepSeek Elastic Compute, DSec.

It consists of three Rust components: an API gateway called Apiserver, per-host Edge agents, and a Watcher cluster monitor. They use a custom RPC protocol and 3FS distributed storage.

The authors report hundreds of thousands of concurrent sandbox instances per production cluster.

This is a scale claim about their platform, not a public benchmark of every substrate's latency or cost.

Four execution substrates

DSec exposes one Python SDK across four forms of execution:

They share command execution, file transfer, and TTY interfaces. A parameter selects the substrate.

One interface reduces the amount of harness-specific work needed to move between tasks. It does not make the underlying isolation, compatibility, and resource behavior identical.

Loading large environment images

For containers, readonly EROFS layers live on 3FS and are mounted into overlay lower directories. Metadata is made available locally, while data blocks are fetched on demand.

For microVMs, overlaybd stores a shared read-only base on 3FS and uses a local copy-on-write layer for writes.

The source describes chainable snapshots and millisecond-scale resumption. That should not be expanded into a claim that every fresh large VM starts in a millisecond under every workload.

On-demand loading avoids fetching an entire image when a task touches only a small portion of it. A task that later reads many files will still pay for those reads.

Packing many sandboxes

At high density, duplicate page caches and runtime contention become important. DSec uses memory reclamation and addresses spinlock contention to improve CPU and memory utilization.

These optimizations matter because agent training can create many lightweight or waiting environments alongside the GPU-heavy model service.

The source gives the direction of the changes, but not a complete public table of per-host density, kernel settings, or resource limits.

Ordered trajectory logs

DSec records commands and their results in a globally ordered trajectory for each sandbox.

After a training task resumes, the client can fast-forward through previously completed calls using cached results. This avoids repeating non-idempotent operations.

The logs also provide provenance and support replay of historical sessions.

Faithful replay of external behavior requires sufficient captured state. A live network endpoint, current time, or external service can change. The report's replay claim describes its platform design; it does not establish that a command transcript alone recreates every arbitrary outside system.

Why these systems affect model quality

Rollout bugs can change the data distribution. Parser bugs can turn a sensible action into a failed task. Lost environment state can make a correct recovery impossible. Numerical differences can make training evaluate a different effective policy from the deployed one.

The model's final quality therefore depends partly on whether the surrounding systems deliver the intended training experience consistently.

A benchmark harness is part of the tested system. It is not a neutral wrapper whose details can always be ignored.

Source: local main.tex, tool schema, interleaved thinking, and Quick Instruction, lines 1126–1216; rollout and DSec, lines 1278–1343. The exponential interruption example is a derivation illustrating the source's length-bias argument.

Continue with benchmark evidence or return to the contents.

18. Reading the Benchmark Evidence

V4's results support a strong model family with clear gains over earlier DeepSeek models. They also show remaining gaps and task-dependent differences.

This chapter reads the numbers in the supplied tables. It does not use the paper's broad ranking language as a substitute for individual results.

What a benchmark score measures

A score belongs to a tested system: model, prompt, reasoning budget, tools, context policy, sampling settings, and grader.

Pass@1 usually estimates the chance that one sampled answer succeeds under the benchmark's rules. Exact match measures agreement with a target answer after the evaluator's processing. Resolved rate measures completed software issues. Elo-like ratings summarize comparative performance under a rating procedure.

Different units should not be averaged without a justified normalization. A 3-point gain in exact match is not directly comparable with a 100-point gain in a rating.

The local source does not provide confidence intervals for most entries. Small differences should therefore be read cautiously.

Base models: a relatively clean comparison

The base-model table uses one internal evaluation framework with consistent settings for V3.2, V4-Flash, and V4-Pro.

This controls more variables than comparisons across unrelated public reports. It still compares complete models that differ in architecture, parameter counts, data, and training.

Selected results are:

The table treats differences of 0.3 or less as the same level.

Flash improves many knowledge and long-context results with substantially fewer active and total parameters. Pro improves most results further, especially knowledge-heavy measures.

The exceptions matter. V3.2 leads both V4 models on BigCodeBench. Flash trails V3.2 on some reasoning and math measures. Flash exceeds Pro on MGSM and CMath.

The accurate conclusion is broad improvement with exceptions, not uniform superiority.

Knowledge after post-training

The large-model comparison uses V4-Pro-Max, Opus-4.6-Max, GPT-5.4-xHigh, Gemini-3.1-Pro-High, K2.6-Thinking, and GLM-5.1-Thinking.

V4 has a large advantage over the listed open baselines on SimpleQA Verified and Chinese-SimpleQA. It remains behind Gemini on both.

GPQA is a useful qualification: K2.6's 90.5 is slightly above V4's 90.1 in the table. The paper's broad statement about leading open models should not erase that row.

HLE combines difficult knowledge and reasoning. A higher score does not cleanly separate factual storage from reasoning skill.

Reasoning and code competitions

The model is particularly strong on the listed code-competition metrics. Difficult math is less uniform: Apex leaves a large gap to the strongest listed closed models, while Apex Shortlist is a stronger result.

A single average would hide that difference.

The Codeforces procedure

The internal evaluation uses 14 Division 1 contests with 114 problems from May through November 2025.

For each problem, the system generates 32 candidate solutions. It samples ten without replacement and randomly orders them as a submission sequence. Tests are constructed by domain experts.

For solved problems, scores follow a penalty scheme based on human participants with the same count of prior failed attempts. Contest scores become ranks and estimated ratings. The reported model rating averages expected contest-level ratings over sampled submission choices and orders.

This is a substantial test-time sampling procedure. It is not equivalent to one answer per problem, a live human contest entry, or an official account rating.

The source's statement that the model ranks 23rd among human candidates should be understood in the context of this evaluation method, not as an independently verified current leaderboard position.

Formal mathematics

The paper evaluates a Lean agent with the compiler and semantic tactic search, up to 500 tool calls, and maximum reasoning effort.

It also evaluates a larger pipeline: generate informal solutions, filter them through self-verification, then use retained solutions to guide formal proofs.

Strict acceptance by the Comparator verifier is required. This strengthens correctness evidence for the formalized statement.

The report states that the scaled Putnam-2025 setting reaches 120/120. It distinguishes this high-compute frontier setting from the practical Putnam-200 Pass@8 setting with bounded sampling and lightweight tools.

These results should not be merged into one cost-free “math score.” Search budget, retries, informal guidance, and formal tools differ.

Formal correctness also applies to the statement actually encoded. The fidelity of informal-to-formal problem translation remains a separate issue in any such evaluation.

Long-context results

At the reported one-million-token setting:

V4 exceeds Gemini on both listed metrics but remains behind Opus. The paper reports stable retrieval within 128K and degradation beyond that, while retaining strong performance at 1M.

MRCR tests retrieval from a large context. CorpusQA is closer to multi-document question answering. Neither establishes exact recall of all arbitrary details.

The authors re-evaluate Opus and Gemini to standardize settings. GPT-5.4 is absent because many long requests failed to return. A blank is not a zero score.

The table's “1M” result should be interpreted with the benchmark protocol. It is not automatically an accuracy measured at every position in a sequence of exactly 1,000,000 tokens.

Agent results

V4 is competitive, with some very close results. It does not lead every open comparison. SWE Pro and HLE with tools show clear room for improvement.

The table also illustrates why adding tools is not a universal ranking-preserving transformation. Different models benefit differently from the harness, search behavior, and interaction budget.

The agent harness

Coding evaluations use an internal harness with Bash and file editing, up to 500 interaction steps, and up to 512K context.

Search-agent evaluations use web search and Python with the same stated step and context maxima. BrowseComp uses a discard-all context-management strategy inherited from V3.2.

Terminal Bench results use the original dataset despite known environment concerns. The report separately gives approximately 72.0 on its Verified subset for Pro.

The original and subset scores should not be substituted for each other. The task sets differ.

Harness details can affect success materially: available tools, message formatting, timeouts, observation truncation, and context retention. The paper standardizes some settings but does not prove that every cross-model result has equal compute or equally tuned prompts.

Flash, Pro, and effort

Selected mode comparisons are:

More reasoning helps many hard tasks. Pro's knowledge advantage remains substantial. Flash's Max mode narrows some reasoning gaps.

The gains are not monotonic everywhere. Flash MMLU-Pro drops from 86.4 in High to 86.2 in Max. Pro MCPAtlas drops from 74.2 to 73.6. Such changes may reflect noise or real trade-offs; no intervals are given.

On Flash Terminal Bench, Max gains only 0.3 over High in the table. That suggests measuring actual cost before choosing Max for every coding request.

Missing entries and uncertainty

Some K2.6 and GLM-5.1 entries are missing because the authors could not obtain responses. Other blanks have their own evaluation limits.

Missing values prevent complete rankings. They should remain missing.

The report also does not provide full repeated-run uncertainty, contamination analysis for every task, or matched cost curves for all models. These omissions limit strong claims about small differences and causal explanations.

What the evidence supports

The base models show broad improvements. The final Pro model is strong in factual knowledge relative to the listed open baselines, code competitions, long-context processing, and several agent tasks.

The results also show gaps on difficult math, some knowledge tests, and some agent tasks. Larger reasoning budgets help unevenly.

The architectural and systems claims explain why long contexts may be affordable. The benchmarks show that useful task performance survives the aggressive memory design. Neither alone proves universal superiority.

Source: local tables/base_eval.tex, tables/large_eval.tex, and tables/small_eval.tex; main.tex, Evaluation Setup and Results, lines 1345–1474; formal-reasoning figure caption, lines 1400–1422.

Continue with internal task evaluations or return to the contents.

19. The Internal Task Evaluations

The paper supplements public benchmarks with internal evaluations of Chinese writing, search, professional work, and coding.

These tasks can resemble actual use more closely than short benchmark questions. Their interpretation depends heavily on task selection, grading, and the populations represented.

Pairwise evaluation

Many internal studies compare two model outputs for the same request. A judge assigns a win, a loss, or a tie.

For $N$ examples,

A non-loss rate includes ties:

Neither is the same as an absolute task-success rate. One model can win a comparison even if both responses are flawed.

Tie handling matters. Removing ties changes the denominator and can make a result look much larger.

Chinese functional writing

The study compares V4-Pro with Gemini-3.1-Pro on 3,170 functional-writing prompts.

The total row gives:

V4 leads across the broad categories shown in the table. Its advantage is smaller in some subcategories, including advertising copy and some formal documents, and stronger in email, technical text, and several reflective or explanatory tasks.

The source attributes some Gemini losses to stylistic preferences overriding explicit user requirements.

That explanation is an author interpretation of evaluated cases. The table alone measures comparative outcomes, not the causal reason for every loss.

Uneven subcategory sizes

The functional-writing table includes 527 reports but only one example in one congratulatory-text subcategory.

A 100% rate on one example and a 66% rate on hundreds of examples have very different evidential weight.

The overall result is weighted by the collected sample mixture. If an application mostly handles short advertising copy, the total functional-writing rate may be less relevant than the advertising subset.

A task distribution should be compared with the intended user population before generalizing.

Creative writing: two axes

Creative writing is evaluated separately for instruction following and writing quality.

The total row reports 2,837 judged outcomes:

The difference between axes is useful. A response can be judged well written while missing a constraint. A response can satisfy formal instructions yet be stylistically weak.

Lyrics are a notable exception in instruction following: the table gives V4 26.67% and Gemini 73.33% across 30 examples. Broad creative-writing success does not imply an advantage for every genre.

A source-table inconsistency

Three displayed sample counts do not equal their outcome sums:

The displayed sample-count column sums to 2,846, while both sets of outcome counts sum to 2,837. The total row uses 2,837.

The reported aggregate percentages match the outcome total. This book preserves the reported aggregate result and flags the discrepancy. It does not silently change the three rows or assume a reason for the nine-example difference.

Possible explanations include a table error or unreported exclusions, but the local source does not settle which.

Complex instructions and multi-turn writing

Against Claude Opus 4.5, V4-Pro has:

The total rates are 45.9% for V4 and 52.0% for Opus, with about 2.0% ties.

This qualifies the larger writing studies. V4 can lead on the collected everyday and creative prompts while trailing on a smaller set chosen for complex constraints and multiple turns.

The right question is which task population a user faces, not which headline uses the largest percentage.

Retrieval-augmented search

The chatbot's non-think search path uses retrieval-augmented generation: retrieve material, then answer from it.

The internal study compares V4-Pro with V3.2 across 956 search Q&A examples.

Most cases tie. V4 wins substantially more often than it loses among the remaining cases.

Single-value factual lookup and planning show relatively strong gains. Comparison and recommendation have smaller margins.

The study supports improvement within this search pipeline. It does not show that V4 answers 89.6% of all search requests correctly; wins plus ties against a baseline are not an accuracy measure.

Agentic search versus RAG

Agentic search lets the model issue several search and fetch actions, inspect results, and continue.

Across 869 examples, the table reports:

The harder subjective questions show the strongest agentic win rate, 68.5%.

The comparison changes the workflow, not just the model weights. A more adaptive retrieval process can obtain evidence that a single retrieval stage misses.

Search cost needs more than token counts

The mean cost table gives:

The agentic path uses about 30.6% more prefill tokens and 16.7% more output tokens by these means. The source says most tool calls are parallel.

These counts do not establish a complete dollar or latency comparison. Search-provider costs, fetching, cache hits, model billing conventions, and tool wait times also matter.

Calling the cost increase marginal is an author judgment based on their deployment. A user should measure it in their own pipeline.

Chinese professional tasks

The paper describes 30 advanced Chinese professional tasks across 13 industries. Tasks include analysis, document generation, and editing. The harness provides Bash and web search.

Human annotators blindly assess V4-Pro-Max against Opus-4.6-Max on task completion, instruction following, content quality, and formatting aesthetics.

The paper reports a 63% non-loss rate for V4.

This includes ties and comes from a small task set. It is evidence about those collected workflows, not a measured success probability for all professional work.

Blind grading is a useful control. The source does not provide every rater-agreement statistic, scoring detail, or repeated-sampling result.

Reported strengths and weaknesses in professional work

The authors describe strengths in completion, content quality, long-form writing, and Chinese formal conventions.

They also report missed formatting constraints, weaker concise summarization, and room for improvement in slide aesthetics.

These observations are useful because they identify concrete failure modes. A model can produce a detailed report while failing a requested page limit or visual layout.

The generated examples shown in figures are selected illustrations. They should not replace the full evaluation distribution.

Internal R&D coding tasks

The team collects roughly 200 tasks from more than 50 engineers, then retains 30 after filtering. The tasks span feature work, bug fixes, refactoring, and diagnostics in stacks including PyTorch, CUDA, Rust, and C++.

Each includes a repository, execution environment, and human scoring rubric.

The reported pass rates are:

On 30 tasks, one task is roughly 3.33 percentage points. Rounded rates and small samples make close differences uncertain.

The benchmark is valuable because it uses real internal work. The selection and environment also make it specific to that engineering population. No complete public task set or confidence interval is supplied here.

The developer survey

Among 85 internal users, 52% say V4-Pro is ready to be their default coding model, 39% lean yes, and fewer than 9% say no.

This measures stated acceptance among experienced users in the team. It is not an objective task-success metric or a survey of all developers.

Respondents also mention trivial errors, misunderstanding vague prompts, and occasional overthinking. Those complaints fit the pattern that strong benchmark performance can coexist with frustrating everyday mistakes.

What these studies add

The internal studies expand the task range and reveal user-facing behavior that public benchmarks may miss. They also show that the model's advantage depends on task type.

Their main limits are private task selection, unequal subgroup sizes, small samples in some studies, and incomplete uncertainty reporting.

The useful conclusion is specific: the model performs well on the team's collected writing, search, professional, and coding workflows, with identifiable exceptions. Generalizing further requires a matched evaluation on the intended workload.

Source: local main.tex, Performance on Real-World Tasks, lines 1476–1576; tables/funcwrite.tex, tables/rag_eval.tex, tables/search.tex, and tables/code_agent_dsbench.tex. Percentage increases, row-sum checks, and small-sample arithmetic are derived.

Continue with limits and practical use or return to the contents.

20. Limits and Practical Decisions

The paper supports an important result: aggressive sequence compression can coexist with strong language, reasoning, and agent performance in a model trained for very long contexts.

It does not establish that every design choice is necessary, that every long-context detail is recoverable, or that every deployment will achieve the estimated cost ratios.

This chapter turns the preceding mechanisms and evidence into practical questions.

The strongest supported claims

The architecture reduces the number and size of historical memory entries. The source gives enough dimensions to explain why cache growth can be far smaller than a conventional GQA baseline.

The reported benchmark results show useful performance after that compression. The base-model table also shows broad gains before post-training.

The systems work explains how the team handles the architecture's irregular operations: expert routing, multiple cache types, matrix-shaped optimizer updates, and long trajectories.

The post-training design gives a concrete route from domain specialists to one model, using full-vocabulary teacher feedback on student-generated histories.

These claims fit together. Lower history cost makes long tasks more feasible; training teaches the model to use the compressed representations; serving and rollout systems make that training and use workable.

What is not isolated

V4 changes architecture, data, training scale, optimization, post-training, and systems together.

Without controlled ablations, a final score cannot assign a precise share of improvement to CSA, HCA, mHC, Muon, data curation, or OPD.

A larger Pro model also has different layer count, hidden width, expert count, top-k, training tokens, and learning rate. Comparing it with Flash does not isolate parameter count alone.

Similarly, a comparison with V3.2 cannot establish the gain from compression alone.

What “million-token intelligence” should mean in use

A large window can preserve more source material and task history. It does not guarantee that the model uses every part equally well.

The reported degradation beyond 128K is a reminder that capacity and effective retrieval differ.

For a document workload, test facts near the beginning, middle, and end; repeated entities; conflicting versions; and questions requiring evidence from distant documents. For a coding agent, test whether it returns to earlier files and decisions without losing constraints.

The relevant question is whether the model succeeds on the dependency patterns your workload requires, at the lengths it actually reaches.

Longer context versus retrieval

A large context and an external retrieval system can complement each other.

Loading everything avoids some retrieval omissions and permits later unanticipated questions. Retrieval reduces prompt size and can provide fresh or more focused evidence.

Compressed attention does not replace the application-level choice of what to include. A model can only use material available through its context and tools.

For a large changing corpus, retrieval may remain necessary even when a single static subset fits. The best design depends on update rate, question distribution, evidence needs, and cost.

The paper does not compare every possible RAG architecture against million-token ingestion.

Choosing Flash or Pro

The reported results suggest a useful starting hypothesis: Pro is more attractive for knowledge-heavy tasks and difficult agent work, while Flash can handle many reasoning tasks with a smaller active and total parameter budget.

This is a hypothesis for workload testing, not a universal purchase recommendation.

Flash still has 284B total parameters. Pro's 49B active count does not make its weight footprint similar to a 49B dense model.

A hosted API can hide deployment complexity. A self-hosted system must account for all weights, precision, network topology, cache capacity, and runtime support.

A decision should compare quality, cost, and latency for the actual request distribution rather than only one competition benchmark.

Choosing reasoning effort

Max improves many difficult tasks but not every task. Some Flash agent results change little from High to Max.

Start by distinguishing routine requests from tasks that benefit from extended search or verification. Measure whether increased effort raises success enough to justify longer completion times and more tokens.

A practical experiment can run the same task set in Non-think, High, and Max, with repeated sampling and identical tools. Record failure modes as well as average scores.

The response's apparent thoroughness should not be the grading criterion. A long explanation can conceal a wrong result or a missed instruction.

Choosing how much history to retain

The paper retains reasoning across user turns in tool-calling scenarios. That can help preserve agent state, but it also preserves earlier mistakes and increases input size.

Applications can keep verified artifacts, concise task-state records, and original evidence alongside model-generated reasoning. These objects serve different purposes.

A tool result is an observation. A model's explanation is an interpretation. A successful test or verifier result is evidence under that check's scope.

Keeping these distinctions clear helps the agent avoid treating its earlier speculation as an established fact.

Hardware and deployment questions

Before estimating capacity, identify: