Introduction

DeepSeek-V3 is best understood as a coordinated design for spending computation well. It combines a large pool of conditional feed-forward networks, a compact attention cache, additional predictive supervision, carefully controlled low precision, and a distributed system designed around the resulting workload. Its final conversational behavior also depends on a substantial data and post-training pipeline.

This book explains the DeepSeek-V3 Technical Report, arXiv 2412.19437v2, revised February 18, 2025. It follows the original V3 model described there. It does not silently incorporate later V3 revisions, DeepSeek-R1 as a separate released model, or newer DeepSeek architectures. Statements about comparative performance refer to the report's evaluation snapshots, not a present-day leaderboard.

The audience is a technically curious reader who wants more than a summary but does not already understand every equation or systems term. The chapters teach the necessary background, derive the main mechanisms, work through numerical examples, and distinguish evidence from interpretation. Each chapter can be read independently; together they move from first principles to a complete model of the system.

The reading contract

Three categories keep the discussion auditable:

• Reported means the paper states a design choice, procedure, or measurement.
• Derived means an equation or numerical calculation follows from stated assumptions. It is not an additional experiment.
• Interpretation means a plausible explanation, engineering implication, or critical judgment. It should not be confused with a demonstrated causal result.

Ordinary explanations of cross-entropy, matrix multiplication, caches, and other established concepts are background. Small toy examples are invented for teaching and are identified as such. The source notes at the end of each chapter name the relevant report sections and local source files.

The requested TeX archive is preserved in the parent directory as 2412.19437v2-source.tar.gz, with its contents extracted alongside it. The main text is main.tex; FP8 details are in content/fp8.tex; benchmark tables are in tables/. Supplementary release material is preserved under reference/. Chapter 27 records the source provenance and important paper/configuration differences.

Contents

Reading paths

Read Chapters 1–3 first if you want the full narrative. Architecture readers can then focus on 4–9 and 12. Training researchers can read 8–20 and 23. Systems engineers can read 3, 5–6, 13–18, and 21–22. Readers evaluating the model for an application can read 1, 3, 12, 21–25, and the integrated request trace in 26.

The recurrence of core concepts is intentional. A reader opening the FP8 chapter should not have to remember the optimizer chapter, and a reader opening the routing chapter should not need to reconstruct MoE from a passing sentence elsewhere.

Conventions and limits

A multiply plus an add counts as two FLOPs. Parameter counts and memory estimates are rounded where stated. Decimal GB means $10^9$ bytes; GiB means $2^{30}$ bytes. For explicit context calculations, this book uses 4K = 4,096, 32K = 32,768, and 128K = 131,072. The paper often uses these labels as rounded descriptions.

The book explains mechanisms in more detail than the report, but additional explanation is not additional disclosure. The exact pre-training corpus, complete training implementation, many RL settings, and full operational traces are not available in the source. Where a reproducible recipe would require them, the chapter says so. Released reference inference code is evidence about an implementation example, not proof of how every training or production kernel works.

Primary source: DeepSeek-V3 Technical Report, version 2. Start with Chapter 1 for the motivation, or use the table to choose a self-contained chapter.

1. The Problem V3 Sets Out to Solve

A language model can become more capable by learning from more data, having more parameters, and receiving better training signals. Each direction also creates costs. Parameters occupy memory. Training requires repeated matrix multiplication and gradient updates. Long prompts create growing attention state. Distributing work across GPUs introduces communication and synchronization.

DeepSeek-V3 is an attempt to improve capability while controlling several of these costs at once. The paper reports a 671-billion-parameter model that activates approximately 37 billion parameters per token, learns from 14.8 trillion pre-training tokens, and uses 2.788 million H800 GPU hours across its reported final training stages. Understanding those three numbers requires understanding what is counted, what work is conditional, and how the hardware executes it. Report, §1.

Capacity is not the same as computation

A dense model applies nearly all of its major layer weights to every token. If it has hundreds of billions of weights, each token incurs correspondingly large matrix work. A Mixture-of-Experts model stores many feed-forward networks but uses a subset for each token. It can therefore expand the collection of learned transformations without multiplying token-level arithmetic by the same factor.

Think of a library with many rooms and a reader visiting a few rooms for each question. The library's total size describes available capacity. The rooms visited describe immediate work. The analogy stops at two important points: experts are learned numerical functions rather than shelves of explicit facts, and the router selects them repeatedly at each layer and token rather than once per request.

V3's 37B active count is a description of its conditional computation. The other weights still need storage, placement, and eventual access. Across a batch of diverse tokens, many experts can be active simultaneously. This is why “a 671B model with 37B active” is neither a dense 37B model nor a free way to ignore 634B weights.

Why one efficiency trick is insufficient

Suppose conditional experts reduced feed-forward arithmetic perfectly. The model could still be impractical if expert dispatch saturated the network. Suppose the network moved tokens instantly. Long-context attention could still exhaust memory. Suppose attention state were compact. Training could still require large optimizer states and saved activations. Suppose every tensor were compressed. Numerical error could still destroy learning.

V3 addresses these problems through linked choices:

The central research story is the interaction between these rows. Smaller experts are useful only if their communication is affordable. FP8 helps both matrix throughput and activation traffic. Memory savings allow training without tensor parallelism, removing a separate communication burden. Batch-wise balancing permits specialization, while deployment redundancy compensates when serving traffic differs from training traffic.

Training and serving have different objectives

Training processes large batches and revisits the same global architecture many times. Its objective is usually aggregate progress per unit cost: useful tokens learned per second, stable optimization, and successful completion. A training step cannot simply omit inconvenient tokens without changing the learning process.

Serving receives irregular requests with different lengths and deadlines. A user cares about time to the first token and delay between generated tokens. An operator also cares about aggregate throughput and hardware occupancy. These goals can conflict: waiting to form a large batch improves GPU utilization but increases queueing delay.

V3's report separates prefill, which processes an existing prompt in parallel, from decode, which produces new tokens sequentially. It uses different expert placement and parallelism for each. This follows from the workload rather than from a purely architectural preference. Large prompt batches favor compute-efficient expert GEMMs. Small decode batches make weight movement and latency more important.

What is actually new in V3

The paper does not invent the Transformer, conditional experts, low-rank representations, or reinforcement learning. It explicitly builds on DeepSeek-V2's MLA and DeepSeekMoE. Its prominent V3 additions include the routing-bias balancing strategy, sequential multi-token prediction training, a large-scale validated FP8 recipe, and the systems work needed to make the complete model economical.

This distinction helps interpret novelty. A mechanism may be inherited and still be essential to the final result. A systems contribution may not change the mathematical hypothesis class but may enable a larger or longer run within a budget. A better post-training pipeline may change observed task behavior without changing the core Transformer block.

Conversely, a good final benchmark does not establish how much each component contributed. Controlled ablations are needed for causal attribution. The report supplies useful smaller-scale comparisons for MTP, balancing, and FP8. It does not provide a factorial experiment removing every component from the full 671B run.

The meaning of the cost headline

The reported $5.576M is calculated by multiplying 2.788M GPU hours by an assumed rental price of$2 per H800 GPU hour. It is an estimate for the final reported training stages. The paper explicitly excludes earlier architecture, algorithm, and data research and ablation costs.

A GPU hour is a resource unit, not an elapsed hour. Two thousand GPUs operating for one hour consume approximately two thousand GPU hours. Under continuous full-cluster use, the reported pre-training budget of 2.664M GPU hours corresponds to roughly 54.2 elapsed days on 2,048 GPUs. This is arithmetic, not an independently observed schedule.

The figure is therefore useful for comparing a stated final-run resource budget. It cannot by itself answer the total cost of creating DeepSeek's research organization, procuring infrastructure, collecting data, conducting unsuccessful experiments, or running the deployed service. Chapter 25 develops this accounting carefully.

What the model is and what the paper measures

The original V3 is a text language model. Its base checkpoint learns a distribution over continuations. Its post-trained checkpoint is optimized to answer instructions and follow preferences. The report studies both, with different benchmark protocols.

An assistant's answer combines at least three influences: knowledge and representations learned in pre-training, behaviors induced by post-training, and the inference procedure used for that request. A correct mathematical answer can depend on all three. A wrong factual answer can occur even when the model excels at code generation. An impressive judge preference score can coexist with imperfect factual reliability.

This book treats capability as a profile rather than a single scalar. It asks which tasks improve, under what prompts and output budgets, and with what measurement uncertainty. The report's historical comparisons with specific GPT-4o and Claude versions should be read at that level of specificity.

A useful question to carry forward

For every component, ask: Which scarce resource does this choice save, and what new constraint does it introduce? MLA saves persistent attention state but requires specialized execution to realize the benefit. MoE saves per-token dense work but introduces irregular routing. FP8 saves bytes and accelerates multiplication but requires deliberate control of numerical error. Distillation improves the training examples but inherits the strengths and biases of its teachers.

That question connects the chapters. The next chapter supplies the common vocabulary needed to follow the answers.

Source notes and further reading

Reported model counts, token budget, costs, and contribution claims come from main.tex, lines 151–267, especially the cost table and its accompanying qualification. See the report's introduction. Deployment distinctions come from §3.4 and limitations from §6. The resource tradeoffs and library analogy are explanatory interpretations.

Continue to Chapter 2: Foundations or Chapter 3: Model map.

2. The Foundations the Paper Assumes

DeepSeek-V3 is a decoder-only autoregressive Transformer with conditional feed-forward layers. That phrase packs together several independent ideas. This chapter unfolds them so the later architecture and infrastructure discussions have a common foundation.

You need only elementary algebra and the idea that a vector is an ordered list of numbers. A matrix is a learned linear transformation between vectors. More advanced equations later in the book are explained in terms of these operations.

From text to token IDs

A tokenizer converts text into a sequence of integers. A token might represent a word, a word fragment, punctuation, whitespace, or part of a character's byte encoding. Tokens are not universally comparable units of language: different tokenizers can encode the same text in different numbers of tokens.

Let a sequence be $x_1,x_2,\ldots,x_T$, with each $x_t$ drawn from a vocabulary of size $V$. An embedding table maps token ID $x_t$ to a learned vector $h_t\in\mathbb{R}^d$. The hidden width $d$ is the number of numerical features carried through most of the model. In V3, $d=7168$.

The embedding is not a fixed semantic definition. Training changes it, and the subsequent layers contextualize it. The representation of a token inside a sentence can therefore differ substantially from its initial table lookup.

Autoregression and probability

An autoregressive model decomposes a sequence probability as

Each term is a distribution over the next token given earlier tokens. This decomposition gives both a training objective and a generation procedure.

During generation, the model evaluates the distribution, chooses or samples a token, appends it to the context, and repeats. New positions depend on previous choices, so ordinary generation is sequential. During training, the full observed sequence is available, and the model can evaluate next-token predictions for many positions in parallel while masking access to future tokens.

This distinction resolves a common confusion: parallel training does not imply that normal inference can generate an entire unknown answer in one pass.

Logits, softmax, and cross-entropy

The final hidden vector is projected into $V$ numbers called logits. Softmax converts them into probabilities:

A large logit relative to the others gives a large probability. Adding the same constant to every logit changes none of the probabilities, which permits numerically stable implementations that subtract the maximum logit before exponentiation.

For a true next token $y$, the cross-entropy loss is $-\log p_y$. If the model assigns probability 0.5 to the correct token, the loss is about 0.693 nats. If it assigns 0.01, the loss is about 4.605. Lower loss means the observed data receive greater probability, averaged across the evaluated positions.

A lower language-model loss does not automatically mean more helpful answers. The loss rewards prediction of the training distribution, which can include text that is unhelpful, false, or unlike an assistant response. Instruction tuning and reinforcement learning target additional behavior.

What a Transformer block does

A simplified pre-normalized decoder block has two stages:

Attention exchanges information among positions. The feed-forward network transforms each position's representation independently, after attention has supplied contextual information. Residual addition preserves a direct path through the block. Normalization helps control the numerical scale of the representation.

V3 replaces ordinary attention with MLA and replaces the feed-forward networks after the first three layers with MoE. It still uses this broad pattern of contextual mixing followed by a token-local transformation.

Do not interpret a layer as a rigid human reasoning step. The model learns distributed features. A single concept can be represented across many dimensions and layers, and one layer can contribute to many behaviors.

RMSNorm and SwiGLU

RMSNorm divides a vector by its root mean square and then applies learned scaling:

It does not subtract the mean, unlike standard LayerNorm. The small $\epsilon$ prevents division by zero or extremely small denominators. Normalization is one of the operations the FP8 recipe keeps at higher precision.

A SwiGLU feed-forward network uses a nonlinear gating branch:

The symbol $\odot$ means elementwise multiplication. If $h$ has width $d$ and the intermediate branches have width $f$, the three main matrices contain approximately $3df$ weights. This simple count becomes useful when accounting for hundreds of V3 experts.

Attention heads and caches

Attention computes queries, keys, and values. A query represents what the current position is looking for; a key helps determine how relevant another position is; a value carries information to combine. These are learned projections, not explicit natural-language questions or fact records.

Multiple heads use different learned projections. Each can attend differently, and their outputs are combined. Causal masking prevents a position from reading later positions.

During decode, the keys and values for previous tokens need not be recomputed from scratch. A KV cache stores reusable attention state for each previous token at each layer. The cache grows with sequence length, number of simultaneous sequences, and the amount of state stored per token. MLA changes the last factor dramatically.

Cached state is separate from model weights. Weights are learned and shared across requests. A cache belongs to a particular input history and must reflect the exact sequence and model configuration that produced it.

Backpropagation and optimizer state

Training first computes predictions and loss in a forward pass. Backpropagation then computes how the loss changes with each learned weight. The resulting gradients drive an optimizer update.

For a matrix operation $Y=XW$, the backward pass needs gradients for both $X$ and $W$. These involve matrix multiplications with transposed operands. This is why reducing the precision of only the forward pass is not the whole story: training executes forward, input-gradient, and weight-gradient GEMMs.

AdamW maintains running first and second moments of gradients. These are additional arrays associated with the parameters. Training also needs gradients, master weights, and saved activations. A checkpoint's inference weight size therefore understates the memory needed to train it.

An optimizer step is not necessarily one forward micro-batch. Several micro-batches can accumulate gradients before the weights are updated. Distributed training adds synchronization and sharding around this process.

Compute, bandwidth, and latency

GPU performance has at least three relevant ceilings. Arithmetic throughput limits how fast matrix operations can execute. Memory bandwidth limits how fast weights and activations can be read or written. Communication bandwidth and latency limit movement between GPUs.

Arithmetic intensity is the ratio of useful operations to bytes moved. A large matrix batch can reuse weights across many tokens, producing high arithmetic intensity. A single-token matrix-vector operation often reads many weights for little reuse, making memory traffic more important.

A rough roofline bound is

It is a bound, not a runtime predictor. Kernel overhead, layouts, synchronization, occupancy, and interference can make real performance lower.

This explains why V3's prefill and decode systems differ. Prefill has many available token positions. Decode provides one new position per active sequence per step. Conditional experts further split those positions into smaller groups.

What to remember before proceeding

A model weight, a token activation, a gradient, an optimizer moment, and a KV-cache entry are different objects with different lifetimes. A training token, an optimizer step, a GPU hour, and a user request are different units. A FLOP count, an active-parameter count, and elapsed latency measure different things.

Keeping those distinctions clear is enough to follow most of the report. The next chapter puts actual V3 dimensions on this conceptual skeleton.

Source notes and further reading

V3's use of the Transformer framework, MLA, MoE, and dimensions is reported in §2 and §4.2, corresponding to main.tex lines 270–455 and 714–744. The elementary equations and roofline discussion are background derivations; they are not claims of additional V3 measurements. V3's precision assignments are detailed in content/fp8.tex.

Continue to Chapter 3: Model map.

3. A Complete Map of the Model

DeepSeek-V3's main model has 61 Transformer layers and a hidden width of 7,168. Every layer uses MLA. The first three layers use dense feed-forward networks; the remaining 58 use a shared expert plus selected routed experts. An additional MTP module supplies an extra future-token training objective and can optionally participate in speculative decoding.

This chapter gives a structural map and an approximate parameter inventory. A parameter count is easiest to understand when it is built from the matrices that implement the model, rather than treated as an opaque headline.

Original illustration from DeepSeek-AI, report §2. The hatched vectors are cached during inference. The diagram presents the general block; the first three V3 feed-forward layers are dense.

The configuration at a glance

These values are stated in §4.2. The release configuration supplies additional details, such as a dense intermediate width of 18,432 and exact vocabulary size of 129,280. Those are supplementary implementation facts, not values stated at that precision in the paper.

Following a token through the stack

A token ID becomes a 7,168-element embedding. Each layer normalizes its current representation, constructs attention projections, mixes information from earlier positions, and adds the result through a residual path. The feed-forward stage then applies either a dense SwiGLU network or the MoE transformation.

In an MoE layer, every token passes through the shared expert and eight selected routed experts. Their outputs are combined, with routed outputs weighted by the router. The selected set can change for the next token and for the next layer.

After the final layer and normalization, the output projection produces logits over the vocabulary. Sampling or greedy selection chooses the next token. The attention state at every layer is retained for subsequent decode steps.

This is the ordinary main-model inference path. The MTP module does not have to be executed to obtain a valid next-token distribution. Its training influence is already present in the learned main-model weights.

Counting one expert

Ignoring biases and small normalization vectors, a SwiGLU expert has three matrices. With $d=7168$ and $f=2048$, its main weight count is

That is about 44.04M parameters per expert. A layer stores 257 experts when the shared expert is included, so its expert matrices contain approximately

Across 58 MoE layers, this becomes approximately 656.94B. The routed-expert pool dominates the total model count.

For one token, only nine experts are evaluated per MoE layer. The corresponding matrix count across the 58 layers is

This figure is the active expert portion, not the whole active model. Attention, dense early layers, embeddings/output projection, and routing also contribute.

Counting MLA approximately

Use the dimensions above and consider the major projection matrices. A useful approximate inventory for one layer is:

The total is approximately 187.10M per layer, or 11.41B across 61 layers. The separate key RoPE projection and joint KV projection can be represented as a combined matrix in an implementation without changing this inventory.

This count also shows why the residual width need not equal heads times per-head width. The concatenated attention value output has 16,384 components, which the output projection maps back to 7,168. “128 heads of width 128” is therefore compatible with a 7,168-wide residual stream.

Completing the approximate total

Using the released dense intermediate width, each early dense FFN contains

weights. Three such layers contribute about 1.19B. The separate embedding and output matrices each contain $129280\times7168\approx0.927$B weights, for about 1.85B combined. The released configuration says they are not tied to each other.

Add the main pieces:

Small routing matrices and normalization parameters fill in the remaining details. The result is consistent with the paper's rounded 671B claim.

Likewise, replacing the stored expert count by the active expert count gives roughly

This explains the rounded 37B headline. It is a principal-matrix accounting convention, not a claim that every scalar counted performs an identical amount of arithmetic for every token. In particular, an embedding lookup reads a row rather than multiplying the complete embedding table by a dense vector.

Why the downloadable weights can be larger

The official repository describes the downloadable model as approximately 685B parameters: 671B for the main model and 14B for the MTP module. The module contains an additional Transformer block and a projection that combines main-model state with a shifted token embedding.

MTP shares the main embedding module and the main output head. This does not mean the embedding table and output matrix are tied to each other. There are two separate notions of sharing: sharing a module across the main model and MTP, and tying input/output weights within the main model. V3 uses the former; the release configuration indicates it does not use the latter.

The 671B and 685B figures describe different scopes in the repository headline. The more detailed weight note separately lists about 11.5B unique MTP parameters plus shared embedding and output modules, and gives a main-model active count of 36.7B. Those rounded inventories do not exactly match every headline or the approximate matrix sum above. This book preserves the differences rather than claiming an exact unique-parameter reconciliation; Chapter 27 records them. Chapter 9 explains the extra module's computation.

Configuration values are not all behavioral guarantees

The paper says 128K context. The inspected release configuration has max_position_embeddings: 163840, while the official model table advertises 128K. An allocation or position-limit field alone does not establish tested comprehension quality up to that larger value. This book uses the report's 128K scope for capability discussion.

The report states a training initialization standard deviation of 0.006; the release configuration's initializer_range is 0.02. A loaded checkpoint already contains weights, so a library initialization default need not describe the historical training run. Treat these as different pieces of evidence rather than forcing them into a single recipe.

The release configuration also includes a routed scaling factor of 2.5. The paper's schematic MoE equations emphasize normalized affinities without spelling out every inherited width scaling detail. Exact implementation claims should therefore be based on the pinned example or configuration and labeled accordingly.

What this map enables

The map allows a reader to answer three recurring questions. Most stored capacity is in expert matrices. Much of the non-expert active work is in attention projections. The total checkpoint remains large even though each token evaluates a small fraction of the expert pool.

It also makes the systems requirements less mysterious. Expert matrices must be distributed, attention state must be compact, and the training system must store more than inference weights. The later chapters explain how these obligations are met.

Source notes and further reading

Report dimensions: main.tex, lines 714–727, §4.2. MLA equations: lines 294–332. Expert architecture: lines 334–356. Supplementary values: reference/release-config.json and the official weight documentation. All matrix totals above are derived approximations.

Continue to Chapter 4: Attention and position.

4. Attention and Rotary Position

Attention allows a token representation to depend on earlier tokens. Without it, a token-local feed-forward network would process each position independently and could not connect a pronoun to an earlier noun or a code expression to a previous variable declaration.

To understand V3's MLA, first understand ordinary attention and why positional information complicates compression. This chapter derives those ingredients without assuming prior familiarity with attention equations.

Queries, keys, and values

For a hidden vector $h_t$, an attention head constructs three learned projections:

The query at position $t$ is compared with each eligible key $k_j$. A dot product produces a compatibility score. The score is normalized across eligible positions using softmax, and the resulting weights mix their values:

Here $d_k$ is the query/key dimension. The output is a weighted sum, not a discrete retrieval of a single record. Multiple positions can contribute, and each attention head can assign a different distribution.

If scores over three positions are $0,0,\log 2$, their softmax weights are $1/4,1/4,1/2$. This toy example makes the operation concrete: the third value contributes twice the weight of each of the other two, but none is excluded.

Why divide by a square root

Suppose independent query and key components have zero mean and variance near one. Their dot product sums $d_k$ products, so its variance grows roughly with $d_k$. Dividing by $\sqrt{d_k}$ keeps its scale approximately stable as the dimension changes.

This is a heuristic grounded in a simple statistical model, not a theorem about learned attention distributions. Real components can be correlated and have different variances. Nevertheless, the scaling reduces the tendency for wider heads to produce excessively sharp softmax distributions solely because they have more components.

V3 concatenates a 128-dimensional content component and a 64-dimensional positional component, so the basic report equation uses $\sqrt{128+64}=\sqrt{192}$ in the denominator. Long-context YaRN scaling can modify the effective attention scale in an implementation.

Causality is a constraint on information

For next-token training, the state at position $t$ predicts token $t+1$. It must not read token $t+1$ or later. Causal attention implements this by excluding positions $j>t$ or assigning them negative-infinite logits before softmax.

When a whole training sequence is processed in parallel, every row of the attention matrix has its own allowed prefix. Parallel evaluation is therefore compatible with causal information flow. The triangular mask, rather than the order in which GPU instructions happen to execute, enforces the prediction task.

Packed data can introduce a second boundary: two examples may occupy one tensor but be masked so neither attends to the other. The report uses such isolation during SFT but does not use cross-sample attention masking during pre-training packing. Position masking and example masking are distinct controls.

Why position must enter somewhere

A content-only weighted sum does not inherently distinguish an earlier token from a later token with the same representation. Language and code depend strongly on order. “A precedes B” and “B precedes A” cannot be treated as equivalent.

Rotary Position Embedding, RoPE, encodes position by rotating pairs of query and key components. For one pair and one angular frequency $\omega$, position $t$ applies

Different coordinate pairs use different frequencies. The rotated query is $R(t)q_t$ and the rotated key is $R(j)k_j$.

Their dot product has a useful identity:

Because rotations are orthogonal, $R(t)^T=R(-t)$. The score therefore depends on the relative displacement through $R(j-t)$ as well as on the content vectors.

What RoPE does not guarantee

The relative-position identity does not mean the model perfectly understands arbitrary distances. Training exposes it to a distribution of positions and displacements. Longer distances can create phase patterns it has rarely or never learned to interpret.

Nor is RoPE a hard rule that nearby tokens always matter more. Learned content and multiple frequencies can produce varied attention patterns. It gives position-sensitive features that the model can use; it does not directly specify a universal semantic relevance function.

Context extension modifies these frequencies or related scaling so the model can adapt to longer sequences. Chapter 12 examines V3's YaRN procedure.

Why ordinary RoPE obstructs a simple compressed cache

Imagine a low-rank key construction $k_j=U_Kc_j$, where $c_j$ is a compact latent. Without position rotation,

This lets us move the up-projection onto the query and compare directly against the latent cache. A single transformed query can serve all past positions.

If the key is instead rotated as $R(j)U_Kc_j$, the score becomes

The transformation involving $R(j)$ changes with the historical position $j$. It can no longer be absorbed into one position-independent query transformation in the same straightforward way. The issue is the order and dependence of the linear maps, not the fact that rotations are expensive by themselves.

V3 handles this by separating content and position components. The content key can be absorbed algebraically. The positional key is small and stored explicitly.

Decoupled content and position in V3

For head $i$, the full query and key are concatenations:

The content pieces are head-specific. The positional key $k^R_j$ is shared across heads. Each head still has its own positional query.

The score separates cleanly:

The values contain the content information used to produce the output; they are not assigned a parallel RoPE value component in the report's equations. Position influences which values are selected through the score.

A shared positional key does not make all heads identical. Different queries, content projections, and output transformations remain. It means the positional part of historical state is amortized across heads.

Attention is computation over a representation

A KV cache is sometimes casually described as the model's memory. That phrase is useful if interpreted carefully. The cache stores learned projections of each token's evolving representation at each layer, not the original text and not a searchable list of explicit facts.

Attention can connect distant positions, but reading them successfully depends on the learned representations and the query. A token remaining in the cache does not guarantee that the model will use it correctly. Conversely, a compact learned representation need not preserve every detail of the token in a reconstructible form to preserve useful attention behavior.

These distinctions matter when comparing MLA with retrieval systems or summarization. MLA compresses the vector state associated with each token. It does not summarize a group of tokens into one slot or perform external document retrieval.

Source notes and further reading

V3's decoupled query/key equations and attention scale are in main.tex, lines 294–332, §2.1.1. The rotation identity and the three-position example are background derivations. For V3's context extension, see §4.3 and Chapter 12.

Continue to Chapter 5: Deriving MLA.

5. Deriving Multi-head Latent Attention

Multi-head Latent Attention, MLA, replaces separately stored per-head keys and values with a compact joint latent representation. V3 inherits the architecture from DeepSeek-V2. Its purpose is to retain useful multi-head attention behavior while reducing persistent attention state during generation.

The central idea is not simply to store an ordinary cache in fewer bits. MLA changes the learned parameterization of keys and values. Low precision is a separate technique that can additionally reduce bytes.

The dimensions we need

Let $h_t\in\mathbb{R}^{7168}$ be the attention input at token $t$. There are 128 heads. Each has a 128-dimensional content query/key component and a 128-dimensional value component. Its positional query component is 64-dimensional.

The KV latent is 512-dimensional. The query latent is 1,536-dimensional. These bottlenecks are much smaller than the concatenated head projections, which have 16,384 content components across all heads.

For clarity, equations below suppress normalization and some implementation scaling. V3 reports extra RMSNorm after the latent vectors, so actual execution must include it. Matrix absorption operates on the latent after the required normalization; it does not remove the normalization by pretending it is a linear map.

Joint compression of keys and values

MLA first projects the input into a shared content latent:

Head-specific content keys and values are obtained by separate up-projections:

where each $U^K_i$ and $U^V_i$ has shape $128\times512$.

The same latent supplies both key and value branches. This is the meaning of joint KV compression. It allows different learned readouts for matching and content while retaining only one shared content vector per token.

At the linear projection level, the concatenated keys and values are constrained to arise through a common 512-dimensional bottleneck. This is a low-rank parameterization. The overall network remains nonlinear because the inputs come from earlier nonlinear layers, latent normalization is applied, and attention includes softmax.

The decoupled positional key

MLA separately computes a small positional key:

This key is shared across heads. The complete key for head $i$ is $[k^C_{t,i};k_t^R]$. During efficient decode, the model stores $c_t^{KV}$ and $k_t^R$, totaling 576 scalar elements per layer per token.

The word “latent” therefore does not mean that all cached information is one opaque 512-vector. There is also a 64-vector explicitly carrying the positional key. Leaving it out gives the wrong cache accounting and the wrong score equation.

Query compression

Queries are compressed through a different bottleneck:

From this latent, the model generates each head's content query and rotated positional query:

Here $U^Q_i$ has shape $128\times1536$, and $U^R_i$ has shape $64\times1536$.

Unlike historical keys and values, previous queries do not need to remain cached for future decode steps. Query compression is therefore primarily a parameterization and training-activation benefit, not an additional persistent KV-cache factor. Confusing the query latent width with the cached content width leads to substantial memory errors.

The full attention equation

For current position $t$ and historical position $j$, head $i$ assigns a score

Causal softmax over eligible $j$ gives $a_{tji}$. The head output is

An output projection maps the concatenated head outputs back to the residual width:

These equations are enough to implement a straightforward version that expands keys and values. But expanding and caching all heads would forfeit the main memory advantage. The efficient inference path uses the following algebra.

Absorbing the key up-projection

Substitute $k^C_{j,i}=U^K_i c_j^{KV}$ into the content score:

Define a transformed query

Now every historical content score is a dot product between a 512-dimensional transformed query and the cached 512-dimensional latent. No historical content key needs to be stored in its expanded 128-head form.

This identity is exact in real arithmetic for the stated linear maps. Finite precision and quantized implementations can introduce rounding differences, so bit-for-bit equality is not guaranteed across kernel choices. The cache reduction is nevertheless architectural, not a heuristic that throws away additional state after training.

Absorbing the value up-projection

Substitute $v^C_{j,i}=U^V_i c_j^{KV}$ into the head output:

Define $\widetilde o_{t,i}=\sum_j a_{tji}c_j^{KV}$. The model first performs attention-weighted aggregation in the latent space, then applies $U^V_i$ to the result. The up-projection need not be performed separately for every historical token during every decode step.

One can also conceptually combine $U^V_i$ with the corresponding slice of $W^O$. A practical kernel may choose a factorized execution rather than materializing a large fused matrix. The mathematical ability to absorb a map does not uniquely determine its best storage layout.

A small numerical example

Use an invented two-dimensional latent and a one-dimensional value projection $U^V=[2,-1]$. Suppose two cached latents are $c_1=[1,0]^T$ and $c_2=[0,3]^T$, with attention weights 0.25 and 0.75.

Expanding first gives values $v_1=2$ and $v_2=-3$, so the weighted output is $0.25(2)+0.75(-3)=-1.75$.

Aggregating first gives $\widetilde o=0.25c_1+0.75c_2=[0.25,2.25]^T$. Applying the projection yields $2(0.25)-2.25=-1.75$. The order of a shared linear map and weighted summation can be exchanged.

Softmax itself cannot be moved through the projection in this fashion. The score and normalization must still be computed correctly before the weights are used.

What MLA shares and what remains distinct

All heads read the same cached content latent and positional key for a given token. They do not share the same attention distribution. Each head has its own content query transformation and positional query, and its own value readout.

MLA is therefore different from a simplistic statement that “all heads use one key and one value.” The latent is a shared basis from which different head behaviors are constructed. It is also different from grouped-query attention, which reduces the number of independently stored KV heads rather than imposing this joint latent factorization.

Why the cache does not become constant in context length

There is still one latent and one positional key for every token at every layer. Cache size remains linear in sequence length. A current query can still score all prior positions. MLA compresses the channel dimension of stored attention state; it does not remove old tokens, select a sparse subset, or eliminate long-context attention computation.

The distinction is central to V3's limitations. A much smaller linear cache makes long histories affordable, but a million-token context would still raise substantial compute and memory challenges. The V3 report's validated extension is 128K.

Source notes and further reading

The architecture and equations come from main.tex, lines 294–332, §2.1.1. Dimensions and latent RMSNorm are in lines 716–726. Matrix absorption is derived here and corroborated by the absorb branch in the pinned official inference example, class MLA. The example is not the complete production implementation.

Continue to Chapter 6: Cache and attention cost.

6. The Economics of Attention Memory

A long conversation creates a cache at every attention layer. Even when model weights are fixed, this request-specific state can consume much of a serving cluster's memory. MLA reduces the number of stored components per historical token from many head-specific keys and values to a shared latent and a small positional key.

This chapter quantifies that reduction and explains why a cache ratio is not automatically a speedup ratio. It assumes the V3 dimensions: 61 layers, 128 heads, 128-dimensional content keys and values, a 512-dimensional KV latent, and a 64-dimensional positional key.

The basic byte formula

If a cache stores $c$ scalar components per token per layer, uses $b$ bytes per component, has $L$ layers, and retains $S$ tokens, then one sequence requires approximately

This counts the core cached tensors. Allocator padding, page metadata, fragmentation, alignment, temporary attention buffers, and replicated state are separate. A batch with different lengths requires a sum over the sequences, not simply maximum length times batch size unless memory is preallocated that way.

For efficient latent MLA, $c=512+64=576$. At BF16 precision, $b=2$, so each token contributes

across the main model. That is about 70.3KB in decimal units per retained token.

Worked context estimates

These are derived estimates for one sequence across 61 layers. They are not measured process memory. If the cache used one byte per component, its raw tensor portion would be roughly half as large, but scales and representation-specific overhead would need to be included. The report's FP8 training recipe does not by itself establish the deployed KV-cache dtype for every serving implementation.

A 128K request also consumes output-cache space as generation proceeds. If “128K context” is a total supported input-plus-output window, filling the entire window with the prompt leaves little or no room for generation. A service's advertised input and output limits must therefore be read separately from this cache arithmetic.

Comparing against ordinary multi-head attention

A conventional MHA cache with 128 independently stored key heads and value heads, both width 128, stores

components per token per layer. Relative to MLA's 576, the ratio is

At the same layer count, context, and BF16 dtype, that hypothetical MHA cache at 128K would occupy about 523.99GB, or 488.01GiB. This comparison fixes content key/value widths and omits an additional positional key width in the hypothetical MHA baseline. If one cached 192-dimensional keys plus 128-dimensional values, the baseline would be larger still.

The comparison demonstrates why latent compression matters. It does not demonstrate that V3 is 56.9 times faster than an actual competing model. Other models have different numbers of layers, KV heads, head widths, and cache formats.

Comparing against grouped-query attention

Grouped-query attention, GQA, shares keys and values across groups of query heads. If there are $n_{KV}$ stored KV heads of width 128, its content cache has $256n_{KV}$ components per token per layer.

A hypothetical eight-KV-head design therefore stores 2,048 components, about 3.56 times MLA's 576. A one-KV-head multi-query design stores 256 content components, fewer than this MLA configuration. Those are arithmetic comparisons, not quality-equivalent alternatives.

MLA's appeal is not that it is smaller than every possible attention cache. It combines compact shared storage with rich head-specific learned transformations. Whether a particular GQA or MQA model offers a better quality/performance tradeoff requires experiments, not just counts.

Stored state versus attention arithmetic

Ordinary attention reads keys to compute scores and values to produce weighted outputs. With explicit 128-dimensional content keys and values, the core score and value aggregation work per query head per historical token is approximately

where the key score includes the 64-dimensional positional component and multiply-adds count as two FLOPs.

In a straightforward absorbed MLA path, the content score uses a 512-dimensional latent plus the 64-dimensional positional key, and latent output aggregation has width 512. The analogous leading work is approximately

That simple arithmetic is larger, even though persistent state is far smaller. The benefit comes from how the shared latent is stored, reused, and moved, and from avoiding expanded historical KV state. It is not a blanket claim that every inner product becomes shorter.

These estimates omit query transforms, output transforms, softmax, fusion, blockwise algorithms, and hardware utilization. They are useful for seeing the tradeoff but should not be reported as measured end-to-end cost.

Why a kernel can change the result

A naive implementation may dequantize weights repeatedly, expand large KV tensors, or fail to reuse the shared latent across heads. An optimized implementation can tile latent reads, fuse score/value work, avoid materializing full attention matrices, and choose different prefill and decode strategies.

The mathematical formulation permits these choices but does not automatically make them. In particular, an implementation may reconstruct expanded projections transiently during prefill, where larger matrix batches make that attractive, while using latent storage for decode.

The official demonstration code has naive and absorb attention paths. Their existence illustrates that the same model weights can be executed with substantially different memory behavior. An inference engine's actual cache layout must be inspected or measured before using MLA estimates in capacity planning.

Long context is still expensive

For full causal prefill, there are $S(S+1)/2$ eligible query-key pairs per head. The leading pair count therefore grows quadratically with prompt length. Memory-efficient attention algorithms avoid materializing the entire score matrix but do not make all pairwise arithmetic disappear.

For one decode position, attention over history is approximately linear in $S$. A request with a long prompt and long output repeatedly pays for reading a long history. The cache grows as output tokens are added.

MLA improves the storage coefficient. It does not turn full attention into linear-time prefill, constant-time decode, or sparse retrieval. This is why reducing cache memory and increasing the tested context window are related but separate achievements.

Concurrency and service capacity

Suppose, purely as a planning example, an operator has 100GB available for core BF16 latent cache after allocating weights and runtime buffers. Dividing by approximately 9.21GB per full 128K sequence suggests around ten such sequences before overhead. It does not imply that ten can be served within a desired latency target.

The attention state may be sharded or replicated across devices. Expert communication and queueing may become limiting before aggregate cache capacity is exhausted. A batch of short requests can fit far more sequences but create different kernel and routing behavior.

Capacity planning therefore needs two models: a memory budget and a time budget. The memory budget establishes feasibility. The time budget establishes throughput and user experience.

Source notes and further reading

Cached vectors and attention equations: main.tex, lines 294–332, §2.1.1. Dimensions: lines 716–726. All cache sizes, MHA/GQA comparisons, and leading FLOP counts in this chapter are derived under the stated assumptions. For actual serving organization, see Chapter 21.

Continue to Chapter 7: Experts.

7. Fine-grained and Shared Experts

An expert in DeepSeek-V3 is a feed-forward neural network inside a Transformer layer. It is not a separate conversational model, a retrieved document, or an independently prompted agent. The router chooses which expert functions to evaluate for each token representation.

V3 uses this mechanism in 58 of its 61 main layers. Each MoE layer stores 256 routed experts and one shared expert. A token evaluates eight routed experts and the shared expert. This chapter explains how that design changes capacity and computation before Chapter 8 examines balancing and locality.

What one expert computes

A SwiGLU expert maps a hidden vector through two intermediate branches and one output projection:

The input and output width is 7,168; each intermediate branch has width 2,048. The principal matrices contain approximately 44.04M weights per expert.

Under the convention of two FLOPs per multiply-add, applying those matrices to one token costs approximately

The nonlinear activation and elementwise product add work but are small relative to these matrix operations at this width. Routing, dispatch, and output combination add further costs outside the expert itself.

Nine expert evaluations give roughly 792.72M principal-matrix FLOPs per token per MoE layer. Evaluating all 257 experts would instead cost about 22.64B FLOPs. Conditional computation therefore changes the large expert pool's per-token arithmetic dramatically.

Fine-grained experts and combinations

DeepSeekMoE uses many relatively small experts rather than a few very wide ones. This makes it possible to choose combinations of transformations for a token. There are $\binom{256}{8}$ possible routed subsets before locality and learned preferences constrain them.

That combinatorial count is not a measure of model intelligence. Many subsets may rarely occur, experts can learn correlated functions, and selection patterns do not imply independent information. It simply illustrates that choosing eight from 256 offers richer conditional composition than choosing one from a small pool.

At fixed total active intermediate width, splitting work into smaller experts can potentially improve specialization. It also creates smaller matrix batches, more routing metadata, and more communication destinations. The architectural benefit must be paired with a system that handles the resulting irregular work efficiently.

Why have a shared expert

The shared expert is evaluated for every token in the layer. It provides a common transformation that does not depend on top-k routing.

The design motivation is that some features may be useful broadly and need not be duplicated across many routed experts. A shared path can handle common structure while routed paths adapt to conditional needs. The report presents shared-expert isolation as part of the inherited DeepSeekMoE architecture.

This is a motivation rather than a guarantee of a clean semantic division. A shared expert can still learn task-sensitive behavior because its input is contextual. Routed experts can still carry common features. Their roles emerge through optimization, not through a hard assignment of “general knowledge” versus “specialized knowledge.”

The router's affinity scores

For expert $i$, the router computes a scalar from the token's current hidden vector:

The learned vector $e_i$ is called an expert centroid in the report. The affinity $s_i$ lies between zero and one. It is a score for routing and weighting, not a calibrated probability that an expert understands the token.

The sigmoid is applied independently to each expert logit. Its outputs do not sum to one across experts. V3 later normalizes the affinities of the selected experts to construct their mixture weights.

A positive affine dot product gives an affinity above 0.5. But the absolute value alone does not determine selection: other experts, balancing biases, and locality constraints also matter.

Selecting and combining experts

Ignoring balancing and node restrictions for a moment, let $S(u)$ be the selected top-eight set. The normalized mixture weights are

and zero outside the selected set. The report's schematic FFN output is

The residual $u$ remains, and the shared expert is added separately. The routed normalized weights sum to one, but this does not make the entire transformation a convex combination because the shared and residual contributions are also present.

The released configuration includes an additional routed scaling factor of 2.5. The pinned inference example applies this after normalization. Thus an exact implementation equation includes that scaling where applicable; the simpler equation above follows the report's presentation.

A toy mixture

Imagine a four-expert toy model selecting two experts. Their affinities are 0.8, 0.6, 0.3, and 0.1. Without biases or restrictions, experts 1 and 2 are selected, with normalized weights $0.8/1.4\approx0.571$ and $0.6/1.4\approx0.429$.

If their one-dimensional outputs are 10 and -4, the routed contribution is about $0.571(10)+0.429(-4)=4.0$. The shared and residual terms then add their own contributions.

Notice that expert selection and expert output are separate. An expert with a large affinity need not emit a large positive value. Its output is a high-dimensional learned transformation. The router determines where to send work and how to combine it, not the answer directly.

Does top-k routing block learning

The discrete selected set is not smoothly differentiable in the usual sense. Nevertheless, the selected experts receive gradients through their outputs, and the selected affinity weights can receive gradients through the mixture. The network can learn useful routing without differentiating the integer index-selection operation itself.

Unselected experts do not receive the same direct expert-output gradient for that token. If routing collapses onto a small subset, unused experts may learn poorly while selected experts become overloaded. This is both a learning problem and a hardware problem, motivating the balancing controls discussed next.

The report does not provide a full optimizer-level specification of every routing gradient path. A teaching derivation should not be mistaken for the complete training code.

Specialization is statistical, not a job title

An expert can be used disproportionately for code, mathematics, a language, or a structural pattern. But one should not infer that expert 17 is “the Python expert” across all layers. Expert indices are local to layers, and learned features can be much finer or more abstract than human task categories.

A code request can select different experts for indentation, identifiers, comments, and reasoning prose. The same token ID in a different context can take a different route because routing uses the contextual hidden vector.

The paper's specialization study compares expert loads on domains in the Pile test set. Those load profiles support domain-dependent usage patterns. They do not by themselves reveal every expert's internal computation or establish a fully interpretable taxonomy.

Batches change what is active

For one token, eight routed experts are selected in a layer. For a batch, the union of selected experts may approach the whole pool. Expert computation is usually implemented by grouping token assignments by expert and applying batched matrix operations.

A useful thought experiment assumes independent uniform routing, solely to understand scale. After $A$ total expert assignments across the batch, the expected number of distinct experts touched is approximately

Real top-eight routing has dependence within a token and strong learned nonuniformity, so this is not a V3 workload model. It shows why a small active count per token cannot be used as the total batch weight footprint.

Large batches give each expert more token rows and better weight reuse. Small decode batches can leave many expert GEMMs tiny. Performance then depends on memory traffic, communication latency, and efficient placement more than on peak matrix throughput.

The capacity tradeoff

Conditional experts let V3 spend storage on a large collection of functions while limiting per-token arithmetic. They do not eliminate the need for high aggregate device memory. They also introduce a scheduling problem absent from a regular dense FFN: tokens must be regrouped, transported, processed, and restored to their original order.

This is why the expert architecture and the distributed system must be read together. The next chapter explains how V3's router prevents its flexibility from producing an unusable workload.

Source notes and further reading

Expert equations and sigmoid normalization: main.tex, lines 334–356, §2.1.2. Dimensions: lines 716–727. The routed scaling detail is supplementary evidence from reference/release-config.json and class Gate in the pinned official inference code. Numerical mixtures, FLOPs, and occupancy formulas are derived examples.

Continue to Chapter 8: Routing and balance.

8. Routing and Load Balancing

A router must make useful learning decisions and produce a workload that hardware can execute. These objectives are related but not identical. If most tokens choose the same experts, those experts become bottlenecks and the rest remain underused. If every sequence is forced to use every expert equally, useful specialization may be weakened.

DeepSeek-V3's main balancing mechanism adjusts expert-selection biases from observed loads. It also retains a very small sequence-level auxiliary loss. Understanding both parts is necessary to read the phrase “auxiliary-loss-free” accurately.

What conventional auxiliary balancing does

A traditional MoE training objective can include

The balance term rewards a more even distribution of expert assignments or affinity mass. Gradients from this term influence the learned router. A large coefficient can improve hardware balance while pushing representations away from choices favored by the language-model objective.

A small coefficient may have little effect; a large coefficient may overconstrain specialization. Choosing it is therefore a tradeoff. A sequence-level objective can be especially restrictive because it asks each individual document to spread work even when its domain naturally favors some experts.

V3's routing-only bias

V3 computes original affinities $s_{i,t}$ and adds a separate bias $b_i$ for selection:

For an expert that enters $S_t$, the mixture weight still comes from the original affinity:

The bias changes which experts receive work; it is not directly added to the output weighting score. This distinction prevents the load-control state from being interpreted as a learned semantic confidence value.

Selection changes can still affect the model's output, of course. An expert admitted by the bias contributes its learned function. “Only used for routing” means only the selection computation uses the biased score, not that the bias has no behavioral consequence.

The feedback update

At each training step, the system monitors expert load across the batch. It decreases an overloaded expert's bias by $\gamma$ and increases an underloaded expert's bias by $\gamma$.

A schematic controller is

where $\ell_i$ is observed load. This equation summarizes the stated direction of adjustment; the paper does not fully specify every tie, smoothing, or aggregation detail needed for a production implementation.

The reported update speed is 0.001 for the first 14.3T pre-training tokens and zero for the final 500B. Setting it to zero freezes the bias updates during that final interval. It does not remove the existing biases or disable routed experts.

This mechanism resembles feedback control: observe imbalance, nudge future selection, observe again. Its stability depends on the update size, workload, and changing router scores. The paper reports successful balance but does not present a complete control-theoretic convergence proof.

A selection-versus-weighting example

Consider a toy top-two router with affinities $[0.80,0.70,0.60,0.10]$. Without biases it selects experts 1 and 2. Suppose expert 2 has bias -0.20 and expert 3 has bias +0.05. The selection scores become $[0.80,0.50,0.65,0.10]$, so experts 1 and 3 are chosen.

Their mixture weights use 0.80 and 0.60, not 0.80 and 0.65. The weights are approximately 0.571 and 0.429 before any implementation routing-scale factor.

This example also shows that good balance is not free. The controller may select a lower-affinity expert. The aim is a better overall tradeoff between expert learning, task loss, and hardware utilization, rather than maximizing each token's instantaneous affinity ranking in isolation.

The complementary sequence loss

The full V3 training configuration includes

where $N_r=256$, $\alpha=0.0001$, $f_i$ is a normalized frequency of expert selection within a sequence, and $P_i$ is the average normalized affinity mass for that expert.

The report defines

The displayed frequency equation uses the original affinities. It should not silently be rewritten as the exact bias- and node-restricted realized dispatch count without implementation evidence.

Under idealized uniform usage, each $f_i\approx1$ and each $P_i\approx1/N_r$, so the sum is approximately one. Concentrated assignment and mass can make the term larger. Gradients act through the affinity quantities; the hard selection-frequency indicator itself is discrete.

The tiny coefficient is intended to prevent extreme per-sequence imbalance while leaving the batch-wise controller as the main balancing mechanism. Thus V3 is not literally trained with no auxiliary balance loss whatsoever.

Batch balance permits domain specialization

Imagine a batch containing code and prose. Code tokens prefer one subset of experts; prose prefers another. Aggregate load can be balanced even though neither individual sequence is uniform.

Batch-wise balancing allows this distribution. Sequence-wise balancing pushes each sequence toward a more even local mixture and can reduce domain-dependent usage. The report's domain-load figures show stronger specialization patterns in the auxiliary-loss-free comparison model.

The report also supplies an unusually important qualification: at 1B and 3B MoE scales, a batch-wise auxiliary loss matches the validation loss of the bias-based method when balance is comparable. At 1B, losses are 2.258 for sequence-wise loss and 2.253 for both alternatives; at 3B, 2.085 versus 2.080.

This suggests that balancing scope contributes to the observed gains. The evidence does not justify attributing every improvement uniquely to the absence of an auxiliary gradient. Chapter 23 revisits this causal distinction.

Original illustration from DeepSeek-AI, report §4.5.3. These are selected layers of the approximately 16B ablation models with 64 experts, not a direct plot of the full V3 model's 256 routed experts. Stronger peaks show domain-dependent usage, not named semantic roles.

Restricting target nodes

Training distributes experts across eight nodes in a 64-GPU expert-parallel group. A token's eight routed experts are constrained to at most four nodes.

The paper describes choosing nodes using the sum of their highest $K_r/M$ expert affinities. Here $K_r=8$ and $M=4$, so the node score involves its two highest candidate scores. Expert selection then takes place within eligible groups.

The pinned inference example implements eight groups and four selected groups; with correction biases present it sums each group's top two biased scores before masking ineligible groups. This is implementation-level corroboration, with a distinction between the report's general description and the example's exact ordering.

Node selection does not require exactly two final experts on every selected node. It ranks candidate nodes by a top-two summary and then selects experts among the permitted candidates. The final per-node counts can differ.

No token dropping and inference drift

The report states that V3 does not drop tokens during training or inference. In MoE terminology, token dropping means skipping some intended expert processing when capacity is exceeded. It does not mean text truncation or refusing a user request.

Good training balance makes dropping unnecessary in the reported system, but inference traffic can differ sharply from the training mixture. A service dominated by one language or code task can overload a subset of experts despite globally balanced training.

V3 addresses this with redundant expert placement: hot expert weights are replicated, and work is assigned across copies. This preserves the model function more directly than changing semantic routes purely to satisfy runtime load. Chapter 21 explains the deployment details.

Source notes and further reading

Routing biases, complementary loss, and node constraints: main.tex, lines 358–397, §2.1.2. Bias schedule: lines 737–744. Batch-versus-sequence comparisons: lines 912–932, §4.5.3. Exact group-selection example: class Gate in the pinned reference code. The controller expression and numerical example are explanatory derivations.

Continue to Chapter 9: Multi-token prediction.

9. Multi-token Prediction as Training

Ordinary autoregressive training asks each position to predict the next token. V3 adds a second prediction through a sequential Multi-Token Prediction module, MTP. The main model still learns the standard next-token objective; the extra module supplies additional supervision during training.

This chapter explains MTP as a learning method. Its separate use as a draft mechanism for speculative decoding is examined in Chapter 22. Keeping those roles distinct prevents the inference speed claim from obscuring the training contribution.

The ordinary prediction task

At input position $i$, the main model processes the prefix through $x_i$ and produces a representation $h_i^0$. The output head predicts $x_{i+1}$:

Cross-entropy rewards high probability on the actual next token. The full observed sequence provides the targets, but causal masking prevents $h_i^0$ from reading them in advance.

A training objective averaged over positions can be written as $-\sum_i\log p_{i+1}^{0}[x_{i+1}]$, with the appropriate valid-position normalization. Sequence boundaries and padding must be handled consistently.

The V3 additional prediction

V3 uses MTP depth $D=1$. Its extra module combines the main representation at position $i$ with the embedding of the actual next token $x_{i+1}$:

The concatenated input has width $2d$ and the projection $M_1\in\mathbb{R}^{d\times2d}$ brings it back to width $d$. An additional causal Transformer block produces $h_i^1$, and the shared output head predicts $x_{i+2}$:

Thus the second prediction is conditioned on the next token through its embedding. It is not an independent guess at $x_{i+2}$ based only on the prefix ending at $x_i$.

Why this does not violate causality

Take an invented sequence: the, cat, sat, down. At the the position, the main model predicts cat. The additional module combines the state after the with the observed cat embedding to predict sat.

The module predicting sat is allowed to know cat, because cat comes before sat. It is not allowed to know sat or down through another path.

Across MTP positions, the additional Transformer block must preserve the shifted causal alignment. A causal row ending at module position $i$ can use inputs containing ground-truth tokens only through $i+1$ for depth one. This preserves the relevant prefix for predicting $i+2$.

The use of a ground-truth preceding token is teacher forcing, as in standard autoregressive training. Leakage would occur if the prediction path saw its target or later tokens. Merely seeing a known earlier token is not leakage.

General depth and the sequential chain

For a general depth $k$, the module combines $h_i^{k-1}$ with the embedding of $x_{i+k}$ and predicts $x_{i+k+1}$. This maintains a chain of conditional predictions.

Some multi-token approaches instead attach independent heads that predict several offsets from the same representation. V3's sequential design adds the shifted preceding token at each depth and uses a Transformer block per depth. The two designs have different information flow and compute costs.

V3's reported depth is one, so the general formulation should not be read as a claim that the main run used many additional modules. At depth one, the ordinary prediction plus one extra prediction gives two supervised future-token offsets.

Original illustration from DeepSeek-AI, report §2.2. The figure illustrates the general sequential construction with multiple depths; the actual V3 run uses only one additional MTP module.

Loss weighting

The report averages losses over MTP depths and scales the result by $\lambda$:

The extra loss is added to the main training objective. For V3, $\lambda=0.3$ during the first 10T tokens and 0.1 during the remaining 4.8T.

A weight of 0.3 does not mean exactly 30% of every parameter update comes from MTP. Gradient magnitudes and paths differ, and loss normalizations matter. It is a coefficient in the objective. The lower later weight changes the relative training emphasis but does not remove the module.

Near sequence ends, some shifted targets are unavailable. Implementations must mask invalid offsets and normalize consistently. The report's equations state a sequence-level normalization convention; this book does not infer every boundary-handling detail of the training code.

How an auxiliary module improves the main model

The MTP module depends on the main hidden state. Its loss therefore backpropagates through that state into main-model parameters. The main representation is encouraged to support additional future-token prediction, not just the immediate output head's next-token prediction.

The paper suggests two possible benefits: denser training signals and representations that better anticipate future structure. These explanations are plausible. The experiments show downstream gains but do not directly measure “planning” as a distinct internal computation.

An auxiliary module can influence the main model during learning and then be removed during inference, just as an auxiliary classifier can regularize a representation without remaining in the deployed forward path.

What sharing saves

The MTP module uses the main model's embedding module and output head. It has its own combination projection, normalization, and Transformer block. Sharing avoids maintaining separate full vocabulary matrices for every prediction depth.

Physical sharing in a distributed system also requires placement and gradient management. The report places the shallow and deep parts on the same pipeline rank under DualPipe so shared parameters and gradients can be physically reused.

This is separate from tying the main embedding matrix to its output matrix. The inspected release configuration says tie_word_embeddings: false. MTP shares each of those modules with the main path, while those two modules remain distinct from each other.

Training compute is not unchanged

The extra Transformer block, projection, and output-head evaluation cost training computation and memory. MTP is therefore not a free increase in the number of targets.

The report's ablations compare models trained on the same data with the same base architecture, adding a depth-one MTP module to the treatment. They discard the module during evaluation, giving equal main-model inference architecture and cost. This is strong evidence for an inference-time quality benefit from the training procedure.

It is not a matched-training-FLOP comparison. The MTP model receives more work per observed token. A useful follow-up experiment would allocate the baseline equivalent extra compute through more tokens or another training change and compare the resulting tradeoff.

Reading the ablation fairly

At the larger ablation scale, HumanEval rises from 44.5 to 53.7 and GSM8K from 72.3 to 74.0. At the smaller scale, HumanEval rises from 20.7 to 26.8 and GSM8K from 25.4 to 31.4. Many other metrics improve.

There are exceptions. Large-scale MMLU decreases from 67.5 to 66.6, and small-scale NaturalQuestions from 22.7 to 22.3. The result is a broad positive pattern, not a claim that every benchmark improves.

The two baseline scales are approximately 15.7B and 228.7B total parameters. They are not the complete 671B V3 model. Extrapolating the exact point gains to full V3 would go beyond the evidence.

From teaching signal to draft model

During training the preceding shifted token is ground truth. During generation it must be supplied by a generated or proposed token. If the main model emits $x_{i+1}$, the MTP path can condition on its embedding to propose $x_{i+2}$.

That proposal is useful only if the target model verifies it appropriately. Directly emitting the MTP guess without verification would generally change the output distribution. Chapter 22 explains the acceptance rules and why high acceptance can reduce expensive target passes.

Source notes and further reading

MTP formulation: main.tex, lines 410–455, §2.2. Loss schedule: lines 741–744. Ablation table and setup: lines 825–866, §4.5.1. Physical sharing: lines 574–577. The causal example and compute-matching critique are explanatory analysis.

Continue to Chapter 10: Data and tokenization.

10. Data, Tokenization, and Infilling

A model learns from the distribution of text it sees. Architecture determines which functions it can represent efficiently; data determine which patterns training rewards. DeepSeek-V3's 14.8T-token corpus is therefore as important as its 671B-parameter architecture, even though the paper discloses far less about the corpus than about the attention equations.

This chapter separates what is reported from what a reader might wish to infer. It also explains tokenization, document packing, and fill-in-the-middle training, because each changes what a “training example” actually looks like.

What the report says about the corpus

Relative to V2, the paper reports increased mathematical and programming content, broader multilingual coverage beyond English and Chinese, and a refined pipeline intended to reduce redundancy while preserving diversity. The final corpus contains 14.8T tokens measured by the V3 tokenizer.

English and Chinese remain the majority languages. The report does not supply exact proportions, source inventories, a public corpus manifest, all filters, or complete deduplication settings. It also does not disclose enough to reconstruct the exact sample order or reproduce every data intervention.

Consequently, one can explain the architectural recipe in detail without claiming the model is reproducible from this report alone. Data construction is a large unresolved part of reproduction.

Tokens are a measurement choice

Byte-level BPE starts from a byte-oriented representation and learns frequently useful merges into larger units. Its vocabulary can represent text without requiring every word or character to have its own token. Common fragments receive compact encodings; uncommon strings may use several pieces.

The paper calls the vocabulary 128K. The inspected release configuration records 129,280 entries, including the exact inventory used by that configuration. These descriptions differ in precision, not necessarily in underlying intent.

A corpus token count is therefore tied to a tokenizer. If tokenizer A encodes a document in 1,000 tokens and tokenizer B in 800, training on that document contributes different token totals even though the raw text is identical. Comparing token budgets across models without this caveat can misstate the amount of language observed.

More efficient multilingual compression can reduce the number of tokens needed for the same text. It can also change segmentation and the frequency of learning targets. A shorter encoding is not automatically a quality improvement in every setting; the learned model must still use the representation effectively.

Why token boundaries matter

The paper notes a tokenizer change that combines punctuation and line breaks into tokens. Such combinations can encode common formatting efficiently. But a prompt ending just before the expected newline can create a different boundary from the one most often seen during training.

Imagine a few-shot prompt whose repeated answers end with punctuation plus newline as one token. If the final unfinished answer ends at punctuation without a newline, its tokenization context differs from the repeated pattern. The resulting evaluation can be sensitive to a formatting detail that appears semantically unimportant.

The report mitigates this by randomly splitting some combined tokens during training. This exposes the model to alternate boundary cases. The paper does not specify the full split probability and implementation here, so an exact tokenizer-training reproduction would require additional information.

The broader lesson is concrete: benchmark prompts should be stored as exact text, including terminal whitespace. “Same question” is not always “same token sequence.”

Document packing

Training tensors often have a fixed length. Short documents can leave unused space if each occupies a separate sequence. Packing places multiple documents into one training sequence to improve utilization.

The report adopts document packing intended to preserve data integrity, while stating that pre-training does not incorporate cross-sample attention masking. Thus a later packed document can potentially attend to earlier material in the same causal sequence, subject to ordinary token and position handling.

This does not reveal every detail of the packing algorithm. The report does not fully describe boundary tokens, document ordering, treatment of oversized documents, or all masking decisions beyond the stated cross-sample choice.

During SFT the procedure differs: multiple samples are packed, but sample masking keeps them mutually invisible. Pre-training packing and SFT packing should therefore not be described as one identical algorithm.

Why packing can affect learning

Packing changes efficiency because fewer token slots are wasted. It can also change the context seen at sample boundaries. A causal model may encounter transitions between unrelated documents, which are part of the serialized training stream even if boundary markers signal a reset in meaning.

Without cross-sample masking, the model must learn when earlier material is irrelevant. With masking, the tensor may contain several examples while the attention graph treats them as separate histories.

Neither policy can be assessed only by saying it is “cleaner.” The relevant questions are utilization, model quality, boundary behavior, and implementation cost. The V3 paper reports its choices but does not give a full controlled study of packing alternatives in this report.

Fill-in-the-middle training

Ordinary next-token training predicts continuation after a prefix. Editing tasks often need a missing span given both preceding and following text. Fill-in-the-middle, FIM, turns such a task into a causal sequence through reordering.

Split a document into prefix $f_{\mathrm{pre}}$, missing middle $f_{\mathrm{mid}}$, and suffix $f_{\mathrm{suf}}$. V3 uses a Prefix-Suffix-Middle serialization:

<|fim_begin|> prefix <|fim_hole|> suffix <|fim_end|> middle <|eos_token|>

The model generates the middle after seeing both prefix and suffix in the serialized context. The suffix comes later in the original document but earlier in the transformed training sequence. This is compatible with causal attention because causality follows the actual serialized token order.

The report applies FIM at the document level before packing, with rate 0.1. That rate describes its stated transformation policy; it should not be converted automatically into “exactly 10% of all tokens are FIM targets.” Documents and spans vary in length, and sample-rate and token-rate accounting differ.

A code-editing example

Suppose the original text is:

def square(x):
    return x * x

print(square(5))

A teaching FIM split could place the function signature in the prefix, the final print statement in the suffix, and return x * x in the middle. The serialized input gives the model the intended surroundings before asking it to reconstruct the body.

This trains a different conditional problem from simply continuing the function signature. The suffix can reveal how the function is used. Such training is useful for completion and editing, but it does not guarantee that every repository-level edit will be correct. Larger software tasks also require navigation, testing, and understanding constraints outside a local span.

Data quality and duplication

Repeated examples can increase confidence on frequent patterns while reducing the diversity of effective training signals. Deduplication can reduce this concentration, but aggressive filtering can also remove rare useful material.

The paper reports reducing redundancy while retaining diversity; it does not identify a universal optimal deduplication threshold. Any book claiming a specific cleaning algorithm, web-source list, or quality classifier from these few sentences would be inventing details.

Similarly, more math and code can plausibly contribute to strong evaluations in those domains. The final benchmark gains also reflect scale, objectives, post-training, and evaluation choices. Without controlled data-mixture ablations, the exact contribution of the changed proportions is unknown.

Contamination and disclosure

A benchmark can overstate generalization if its questions, answers, or close variants appear in training. The paper's corpus description does not provide enough detail for an external contamination audit of every evaluation set.

This uncertainty is not evidence that contamination occurred. It is a limitation of what can be verified. A rigorous reading distinguishes “not disclosed,” “not independently checked,” and “demonstrated failure.”

Later-dated coding problems help reduce some forms of overlap risk, but timestamps alone do not establish perfect exclusion. Benchmark interpretation still requires the precise training cutoff, source access, and filtering procedure, which are not fully available here.

Source notes and further reading

Corpus, BPE, token-boundary handling, packing, and FIM: main.tex, lines 692–712, §4.1. SFT sample isolation: lines 970–973. Exact released vocabulary count: reference/release-config.json. The examples and contamination discussion are explanatory analysis rather than additional corpus disclosures.

Continue to Chapter 11: Pre-training.

11. The Pre-training Recipe

Pre-training is the long process that turns random weights into a model of the training distribution. V3's main run uses 14.8T tokens at a 4K sequence length, with AdamW, a staged learning-rate schedule, increasing batch size, and changing weights for some auxiliary mechanisms.

This chapter explains the reported recipe and why the controls exist. It does not claim that listing hyperparameters is enough to reproduce the run: data, kernels, distributed ordering, and initialization details also matter.

AdamW and running moments

For a parameter vector $\theta$ and gradient $g_t$, AdamW tracks approximately

After bias correction, the update scales each component by the square root of its second moment and includes decoupled weight decay:

The equation is a conceptual form; exact implementation details include parameter groups and numerical conventions. V3 reports $\beta_1=0.9$, $\beta_2=0.95$, and weight decay 0.1.

The first moment smooths gradient direction. The second moment scales updates relative to recent squared gradients. These arrays increase memory requirements, which the mixed-precision and sharding design addresses.

Learning-rate phases

The main schedule is indexed by consumed tokens, while warmup is stated in steps. These are not interchangeable units because the batch size changes early in training.

The long constant phase supports learning over a large data budget. The decay reduces update magnitude toward the end. The final lower-rate phases refine the checkpoint. These are standard optimization motivations, not separately measured explanations of why this particular schedule outperforms every alternative.

What a cosine decay means

For progress $u$ from zero to one across the decay interval, a conventional cosine interpolation is

At the start it equals $\eta_{\max}$; halfway it equals the mean of the endpoints; at the end it equals $\eta_{\min}$. The report names cosine decay, but a complete reproduction should verify whether its implementation follows this exact parameterization and token accounting.

Warmup and decay serve different purposes. Warmup avoids applying full-sized updates while moment estimates and representations are still forming. Decay reduces late-stage optimization noise. Neither guarantees stability on its own.

Sequence length and batch size

The reported pre-training sequence length is 4K. The batch size grows from 3,072 to 15,360 during the first 469B tokens, then remains at 15,360.

Interpreting the stated batch size as the number of packed 4,096-token sequences, the maximum batch contains

token slots, approximately 62.9M. The initial size would contain about 12.6M slots. These are derived full-length estimates; padding, exact packing, and implementation definitions can affect realized valid-token counts.

Dividing 14.8T by 62.9M gives about 235,240 steps if the entire run had used the maximum full batch. Because early batches are smaller, this is an approximate lower-count reference, not the actual step total. The exact count cannot be recovered without the detailed batch-growth schedule and valid-token accounting.

Why increase the batch

Larger batches reduce gradient noise and improve hardware utilization, but they also change the optimization regime. Early training can benefit from a different noise scale than later training. A gradual increase can trade these effects as the model matures.

For MoE, a larger effective token pool also gives each expert more assignments and makes batch-wise balancing more meaningful. A perfectly balanced enormous global batch can still contain imbalanced micro-batches, so the relationship between the optimizer batch and the dispatched unit matters.

The report states that large expert and data parallelism help address local imbalance. It does not disclose enough to reconstruct every per-expert micro-batch distribution throughout the run.

Gradient clipping

V3 uses a gradient clipping norm of 1.0. In a simplified global-norm formulation, if $\|g\|_2>1$, the gradient is scaled by $1/\|g\|_2$. Its direction is preserved while its magnitude is reduced.

Clipping can prevent a rare large gradient from causing an excessively large update. It does not fix every numerical problem. A NaN gradient, unstable activation propagation, or a systematically biased low-precision operation needs separate handling.

The paper states the threshold but not every distributed implementation detail, such as the exact norm-reduction path across sharded tensors. Those details matter for faithful reproduction.

Other schedules change too

The routing-bias update speed is 0.001 for 14.3T tokens, then zero for the last 500B. The complementary sequence balance coefficient is 0.0001. The MTP loss weight is 0.3 for the first 10T tokens and 0.1 for the last 4.8T.

These schedules show that pre-training is not one unchanging objective and controller. The model receives stronger extra-token supervision earlier and fixed balancing biases during the final refinement interval.

It would be reasonable to hypothesize that freezing biases reduces one source of late-stage variation. The paper does not isolate that choice in a controlled experiment, so this remains an interpretation rather than an established causal explanation.

Initialization and inherited details

The report says learnable parameters are initialized with standard deviation 0.006 and mentions additional RMSNorm after compressed latents plus scaling at width bottlenecks. Minor unspecified architecture settings follow V2.

A modern release configuration may contain a different initialization default. That does not override the report's historical training statement. Loading pretrained weights normally replaces initialized tensors anyway.

Faithful training requires more than reproducing a generic configuration file. It requires the specific initialization, scaling, normalization, optimizer grouping, loss masks, and distributed numerical choices used in the run.

What “stable training” establishes

The authors report no irrecoverable loss spikes and no rollbacks throughout the full training process. This is strong practical evidence that their integrated recipe supported the run.

It is narrower than saying the loss decreased monotonically, that no transient anomalies occurred, or that the system had no hardware interruptions. Those stronger claims are not stated. Training loss normally fluctuates, and operational fault recovery is not identical to rolling model weights back after instability.

The report does not publish full raw training logs, every intermediate checkpoint, or every incident record. An external reader can evaluate the disclosed methods and smaller comparisons but cannot independently reconstruct the complete stability history from the TeX source alone.

The transition out of pre-training

The 14.8T run produces a base model trained mostly on 4K sequences. Two later context-extension phases adapt it to longer contexts. SFT and RL then shape instruction-following and preference behavior.

These stages reuse the learned parameters but change data, sequence length, objective, or sampling process. They should be treated as distinct parts of the recipe. A benchmark of the final chat model reflects the combined pipeline, not just the pre-training schedule.

Source notes and further reading

Optimizer and schedules: main.tex, lines 714–744, §4.2. Stability statements: lines 174–180 and introduction. AdamW equations, token-slot estimates, and step-count bounds are explanatory derivations. Precision choices are in content/fp8.tex and Chapters 16–18.

Continue to Chapter 12: Long context.

12. Extending the Context Window

V3 is primarily pre-trained at a 4K sequence length, then adapted to 32K and 128K through two additional training phases. Extending the context window means more than allocating a larger cache. The model must interpret positional patterns and use information over distances not covered by the main training length.

This chapter explains what changes, what remains expensive, and how to read the evidence. It uses “128K” as the report's tested scope, rather than treating a larger configuration field as an automatic capability guarantee.

Three meanings of a context limit

A model has at least three relevant limits. Its implementation may permit a certain number of positions. Its training may expose it to sequences of a certain length. Its evaluated performance may be reliable only for particular tasks and distances.

These can differ. A position array large enough for 160K tokens does not prove that the model retrieves, reasons, or follows instructions correctly throughout that length. Conversely, a model trained mostly on short examples may generalize to some longer inputs after positional adaptation.

For practical use, the service can impose yet another limit on total input and output tokens. A book about the model should distinguish architectural capacity, learned capability, measured capability, and product restrictions.

Why RoPE extension is needed

RoPE rotates query/key coordinate pairs with position-dependent phases. At larger positions and relative distances, the model encounters phase relationships outside the dominant training distribution.

Naively using the same frequencies at much longer lengths can degrade performance. Simply rescaling every frequency uniformly can also change short-distance structure that the model already learned.

YaRN is a method for extending RoPE-based contexts through a combination of frequency adaptation and attention scaling. The V3 report inherits its detailed configuration approach from V2 rather than reproducing the complete derivation. The exact frequency-by-frequency implementation should be checked against the relevant source or reference code when reproducing it.

Where position lives in MLA

V3 separates a shared positional key from compressed content keys. Its long-context extension is applied in that decoupled RoPE positional pathway. The report describes the YaRN configuration as applying exclusively to the decoupled shared key $k_t^R$.

This should not be interpreted as ignoring the positional queries. A valid score uses compatible query/key rotations. The pinned inference example computes common rotary frequencies and applies them to the positional query and key components. The paper's phrasing identifies the decoupled positional mechanism rather than a modification of the content latent.

The content KV latent remains 512-dimensional. Extending context changes the number and positional interpretation of cached entries, not their per-token content width.

The two adaptation phases

The stated YaRN settings are scale $s=40$, $\alpha=1$, $\beta=32$, and $\sqrt{t}=1+0.1\ln s$. These $\alpha$ and $\beta$ are YaRN parameters, not the sequence-balancing loss coefficient or GRPO's KL coefficient. Symbols recur in different parts of the report with different meanings.

The factor $1+0.1\ln40$ is approximately 1.369. Its exact use in score scaling must follow the implementation convention; the official example modifies the softmax scale with a squared scale factor in its extended-context path.

Why the batch drops by four

Using the explicit lengths 32,768 and 131,072, each phase has the same number of full token slots per step:

The longer sequence length is offset by fewer sequences. Over 1,000 steps, each phase would contain about 62.9B token slots, or about 125.8B combined, under this interpretation.

This is a derived nominal count. It is not an independently stated corpus size or proof that every slot contains a unique token. The report does not disclose enough about the extension data to establish uniqueness, exact repetition, or all valid-token masks.

Equal token counts do not imply equal attention cost. Increasing sequence length at fixed tokens per step increases the number of token-pair interactions per sequence.

The attention-cost consequence

For causal full attention, a sequence of length $S$ has approximately $S^2/2$ eligible query-key pairs. If there are $B$ sequences, leading pair count is approximately $BS^2/2$.

With fixed $BS$, this becomes approximately $(BS)S/2$, proportional to sequence length. The 128K phase can therefore have about four times the leading attention-pair count of the 32K phase at the same nominal token slots per step.

This is a structural estimate, not a prediction that the whole step takes exactly four times as long. Feed-forward work, projection work, communication, kernel efficiency, and sharding contribute too. The reported context-extension budget of 119K GPU hours covers the actual integrated phases rather than this isolated attention count.

The cache consequence

MLA's efficient BF16 cache contains about 70.3KB per retained token across the 61-layer model. A full 128K sequence therefore has about 9.21GB of core latent-plus-position cache, before overhead, compared with about 2.30GB at 32K.

The architecture makes this manageable relative to full per-head MHA storage. It does not make it negligible when many long requests run concurrently. A server also needs weights, temporary tensors, and output growth space.

Training uses different activation lifetimes from serving. A decode-cache estimate is not a complete estimate of long-context training memory.

Needle in a Haystack

The report includes a Needle in a Haystack test after SFT, showing strong retrieval across context lengths up to 128K. Such a test places a target fact or string inside a large distractor context and asks the model to retrieve it.

This is evidence that the positional adaptation and learned attention can preserve access to information across the tested lengths and placements. It addresses an important failure mode: losing a simple distant fact even though the input fits.

It is not a complete test of long-document understanding. A model may retrieve one explicitly queried item while struggling to compare many dispersed claims, follow conflicting instructions, maintain chronological state, or perform a multi-step proof using several distant premises.

Original result figure from DeepSeek-AI, report §4.3. The color scale is the figure's test score from 1 to 10; it should not be relabeled as a universal percentage accuracy. The test is retrieval evidence rather than a complete long-document reasoning evaluation.

Broader long-context evidence

The chat table reports 48.7 on LongBench v2 and 73.3 on FRAMES. These evaluations add tasks involving more complex use of context and knowledge than a single needle retrieval.

They still have specific datasets, prompts, answer formats, and output budgets. The LongBench score is not the probability that an arbitrary 128K document question will be answered correctly. The FRAMES score likewise does not establish universal retrieval-augmented reasoning quality.

Together, the tests support useful long-context capability within the report's protocols. They should motivate application-specific evaluation rather than replace it.

Long context versus retrieval

Putting an entire corpus into context and retrieving selected passages are different strategies. Long context gives the model direct access to more material but pays attention and cache costs over that material. Retrieval reduces input length but can omit useful information or retrieve the wrong passages.

MLA makes larger contexts cheaper in memory. It does not decide that retrieval is unnecessary. For an application, the relevant tradeoff includes document structure, query specificity, repeated use, latency, and the need to verify citations.

A sensible experiment compares full-context prompting with retrieval-based prompting on the same real questions, measuring answer correctness, evidence coverage, cost, and latency. The V3 paper does not supply that universal comparison.

Source notes and further reading

Extension phases and NIAH description: main.tex, lines 746–765, §4.3. LongBench/FRAMES scores: tables/chat_evaluation.tex. Positional implementation: precompute_freqs_cis and class MLA in the pinned official inference example. Token-slot counts, pair-count ratios, and practical retrieval discussion are derived analysis.

Continue to Chapter 13: Distributed training.

13. Distributed Training Fundamentals

Training a 671B-parameter model cannot be understood by imagining one GPU with a very large array. Parameters, activations, gradients, expert assignments, and optimizer state are placed across devices, each with its own communication requirements.

V3 reports a cluster of 2,048 H800 GPUs, eight GPUs per node. GPUs inside a node communicate through NVLink and NVSwitch; nodes communicate through InfiniBand. Its main training configuration uses 16-way pipeline parallelism, 64-way expert parallelism spanning eight nodes, and ZeRO-1 data parallelism. Memory optimizations allow this training design to avoid tensor parallelism.

Data parallelism: different examples, coordinated learning

In ordinary data parallelism, several replicas process different examples using the same model parameters. Their gradients are combined so the update reflects the aggregate batch.

This can increase throughput and effective batch size. It does not, by itself, reduce the memory needed for one full unsharded model replica. Every replica still needs its parameters unless another parallelism or sharding method divides them.

In a model combining pipeline and expert parallelism, “replica” becomes more complicated: a logical copy of the model is itself spread across multiple devices. Data-parallel groups synchronize corresponding portions rather than each GPU storing a complete V3 model.

Pipeline parallelism: divide depth

Pipeline parallelism assigns different layers or chunks to different ranks. An example passes through the early chunk, then its activation is sent to the next, and so on. During backward propagation, gradients flow in the opposite direction.

The model's depth is partitioned rather than every matrix being split. If one stage contains layers 1–4 and another 5–8 in a toy pipeline, neither stage needs the other's main weights. Stage-to-stage activation traffic remains necessary.

A batch is divided into micro-batches so different stages can work concurrently. While stage 2 processes micro-batch A, stage 1 can begin B. Filling and draining the pipeline leaves periods of underutilization called bubbles.

V3 uses 16 pipeline stages with DualPipe's bidirectional scheduling. Because the architecture has different layer types and additional modules, a perfectly equal layer count per rank is not necessarily the best load balance. The report does not disclose a complete per-rank layer placement table.

Expert parallelism: divide the conditional pool

Expert parallelism assigns different experts of a layer to different GPUs. The router groups tokens by selected expert and dispatches them to the owning device. Expert outputs are returned and combined.

With 256 routed experts across 64 GPUs, the simple uniform inventory is four routed experts per GPU for that layer. This does not specify the entire per-GPU model inventory, because a GPU can own multiple layer chunks, shared experts have their own handling, and DualPipe keeps two parameter copies in its schedule.

The expert-parallel group spans eight nodes. Tokens can therefore need cross-node movement even though each token's feed-forward function is local to its hidden representation. The irregular conditional choice turns a token-local mathematical operation into a distributed communication problem.

Tensor parallelism: divide a matrix operation

Tensor parallelism partitions individual matrices or their output/input dimensions across devices. For example, a large output projection can divide its output columns among GPUs, then gather results or sum partial contributions as needed.

This reduces per-device matrix storage and computation but usually introduces collectives inside each layer. Those communications can be expensive, especially across slower links or when repeated frequently.

V3's main training system avoids TP through memory optimization. This does not mean TP is always undesirable or unused by V3. The report's serving architecture uses TP4 for attention, and the official inference demonstration also supports tensor-parallel execution. Training and inference choose different tradeoffs.

ZeRO-1: shard optimizer state

AdamW maintains first and second moments for the parameters. ZeRO-1 shards optimizer states across a data-parallel group so each rank owns only a portion of those arrays.

ZeRO stages describe progressively broader sharding. At a conceptual level, ZeRO-1 shards optimizer state; ZeRO-2 additionally shards gradients; ZeRO-3 additionally shards parameters. V3 reports ZeRO-1, so one should not claim the full run uses ZeRO-3-style parameter sharding merely because it is distributed.

Pipeline and expert parallelism already divide different parameter subsets. ZeRO-1 then reduces optimizer-state duplication within the relevant data-parallel groups. The exact group topology and synchronization order depend on the training framework.

The parallelism sizes are not independent multipliers

A tempting calculation is $16\times64\times DP=2048$, suggesting $DP=2$. But a complete parallel configuration cannot always be reconstructed by multiplying the largest stated group sizes.

Expert and data parallelism can overlap or use different groupings for dense versus expert parameters. A logical expert-parallel group need not be an entirely independent axis of every operation. Shared dense weights and sharded experts can have different synchronization domains.

The product $16\times64=1024$ is compatible with a simple two-copy picture, but the report does not provide enough rank-mapping detail to assert that this is the exact full topology. A faithful implementation needs the actual process-group construction, not just headline degrees.

A token's communication paths

Within one forward pass, a token can trigger several distinct movements:

1. A pipeline transfer carries its hidden representation between layer chunks.
2. An expert dispatch sends copies or grouped slices to the devices owning selected experts.
3. An expert combine returns weighted results to the original token location.
4. Data-parallel synchronization later aggregates relevant gradient information for updates.

These are different operations with different opportunities for overlap. Pipeline traffic follows depth. Expert traffic follows the learned routing graph. Gradient synchronization follows ownership of parameter state.

Confusing them can lead to statements such as “all communication is all-reduce” or “MoE only needs one transfer.” Neither describes this training workload.

Micro-batches and optimizer steps

A global optimizer batch is split into micro-batches for pipeline scheduling. Gradients from multiple micro-batches can accumulate before an update. The system must preserve the intended relationship between forward weights, backward gradients, and optimizer steps.

If a schedule overlaps forward and backward chunks, it does not imply that arbitrary old and new parameter versions can be mixed. Training correctness includes consistent weight versions and appropriate synchronization. DualPipe changes the schedule while maintaining the intended training computation.

Micro-batch size also affects expert efficiency. Smaller units offer more scheduling flexibility but can create smaller per-expert GEMMs. Larger units improve matrix utilization but may increase activation memory and delay pipeline overlap.

Why topology matters

Within-node links and cross-node links have different bandwidth and latency. A routing pattern that is efficient on a fully symmetric interconnect may be expensive on a hierarchical cluster.

V3 limits a token's training destinations to four nodes and exploits fast local forwarding within each target node. This aligns model routing with the physical network. It is a concrete example of co-design: the selection policy and communication kernel are adapted to each other.

This also limits portability of performance claims. Running the same model on a cluster with weaker networking or a different GPU generation can change the bottleneck even when the model equations are unchanged.

What to measure in a distributed run

A useful profile separates stage compute time, dispatch and combine time, pipeline-transfer time, synchronization, idle intervals, and memory peaks. The largest standalone kernel time does not always identify the end-to-end bottleneck because overlap changes the critical path.

Unequal stage work can create idle time even with fast communication. Unequal expert loads can create a long tail even when aggregate bytes fit within bandwidth. Host-side launch overhead can matter when device kernels are short.

The V3 report explains several mechanisms and gives resource costs, but it does not publish every per-rank trace. The next chapter therefore distinguishes its scheduling claims from the conditions required to reproduce them.

Source notes and further reading

Cluster and main parallelism configuration: main.tex, lines 460–476, §3.1–3.2. Memory choices and avoidance of TP: lines 560–577. Serving TP: lines 589–629. DP, PP, TP, ZeRO, and process-group caveats are background explanation.

Continue to Chapter 14: DualPipe.

14. DualPipe and the Critical Path

Communication can consume substantial device and network work without adding the same amount to elapsed step time. This happens when independent useful computation runs while data move. DualPipe is V3's pipeline schedule designed to exploit that possibility in a workload with heavy expert communication.

The report describes an approximate 1:1 computation-to-communication ratio for cross-node expert-parallel training. A serial schedule would spend much of its time moving data. DualPipe reorganizes forward and backward components so a large share of that movement is hidden behind computation.

Work time versus elapsed time

Consider an invented workload with 10ms of computation and 8ms of independent communication. Serial execution takes about 18ms. Perfect overlap could reduce it to about 10ms, plus unavoidable setup and dependencies.

The communication still transfers bytes, occupies network resources, and may use GPU SMs. “Hidden” means it does not fully extend the critical path. It does not mean the operation vanishes or has zero resource cost.

If computation and communication contend for the same memory bandwidth or compute units, overlap can slow both. The ideal $\max(T_{\mathrm{compute}},T_{\mathrm{comm}})$ bound is therefore optimistic. Kernel design must keep interference low enough for the overlap to be useful.

Why pipelines have bubbles

In a simple four-stage pipeline, the first micro-batch reaches stage 1 before stage 4 has anything to do. Later, stage 1 may finish its last forward work while later stages are still processing. Backward dependencies create additional scheduling constraints.

A one-forward-one-backward schedule, 1F1B, alternates forward and backward work when possible. It limits how many activations remain live compared with schedules that run all forwards first, but it still has fill and drain bubbles.

Adding more micro-batches amortizes some idle time, yet also affects activation lifetime and scheduling complexity. A schedule's quality depends on stage count, component times, memory constraints, and the number of micro-batches.

Splitting the backward pass

For $Y=XW$, the backward computation has two main matrix products: one produces the input gradient $dX$; the other produces the weight gradient $dW$.

The input gradient is needed to continue backpropagation into an earlier stage. The weight gradient is needed before the optimizer update but may have more scheduling slack. Separating these operations allows the scheduler to prioritize dependency-critical work and postpone other work into idle intervals.

This is not a mathematical approximation. Both gradients are still computed. The schedule changes when independent pieces execute, subject to correct use of the forward activations and weight version.

DualPipe uses this distinction for attention and MLP backward components, building on the same broad idea as zero-bubble pipeline work.

The components inside a chunk

The report divides a forward chunk into attention, all-to-all dispatch, MLP/expert computation, and all-to-all combine, plus pipeline communication. A backward chunk splits attention and MLP work into input-gradient and weight-gradient parts.

Dependencies prevent arbitrary permutation. Expert computation needs dispatched inputs. Combining needs expert outputs. An earlier layer's backward work needs the relevant input gradient. But work from a different micro-batch or opposite-direction chunk can be available while one path waits for communication.

DualPipe pairs forward and backward chunks and arranges these components to overlap communication and computation. It also tunes the allocation of SMs between communication and compute.

Bidirectional scheduling

DualPipe feeds micro-batches from both ends of the pipeline. The reverse-direction work is arranged symmetrically, giving the schedule more opportunities to pair forward and backward components with complementary needs.

The physical placement involves two parameter copies. This is not the same as reversing the semantic layer order of the model for a token. Each logical example still executes the intended network; the scheduling direction and physical placement describe how its chunks move through ranks.

The report's figure illustrates eight pipeline ranks and twenty micro-batches in two directions. The actual run uses 16-way PP. A diagram of one illustrative schedule should not be mistaken for a complete trace of the production run.

Original scheduling illustration from DeepSeek-AI, report §3.2.1. This example uses eight PP ranks and twenty micro-batches across two directions; the main V3 training configuration uses PP16. Blank regions visualize schedule bubbles.

The reported bubble formulas

The paper compares methods using $PP$ for pipeline degree, $F$ for forward chunk time, $B$ for full backward chunk time, $W$ for weight-gradient time, and $F\&B$ for a mutually overlapped forward/backward pair:

The activation column is the paper's normalized scheduling measure, not literal bytes. For $PP=16$, moving from 16 to 17 represents a 6.25% increase in that measure. Actual memory also depends on each chunk's tensor sizes and storage policy.

The bubble expressions assume the relevant schedule model and component timings. They are not universal runtime formulas applicable to arbitrary imbalanced stages.

Why the two copies may be affordable

Duplicating parameters sounds expensive. In a dense model with every stage storing large full matrices, it can be. V3 shards its large expert pools across many devices and reduces other memory through FP8 storage, recomputation, and optimizer choices.

The report argues that large EP makes the extra parameter copy manageable. This is a combined systems claim: DualPipe's feasibility depends on the memory design, not on its scheduling formula alone.

MTP's shared embedding and output head also benefit from the placement of shallow and deep parts on the same pipeline rank. Logical sharing becomes physical sharing rather than maintaining unnecessary remote duplicates.

Conditions for hiding communication

Overlap works when enough independent computation exists, communication completes before its result is needed, and shared-resource contention is controlled. It can fail when expert load becomes skewed, messages become too small, a stage has too little compute, or network traffic exceeds sustained bandwidth.

As model scale changes, the report's near-zero exposed all-to-all claim is conditional on maintaining a favorable computation-to-communication ratio. It should not be generalized to every MoE model, every batch size, or every network.

Increasing expert count can change message patterns even if active arithmetic stays similar. Increasing context can change attention's share of the stage. Increasing micro-batch size can improve expert GEMMs while changing live-activation requirements. These changes need profiling and retuning.

A toy critical-path comparison

Suppose forward attention takes 3ms, dispatch 4ms, experts 5ms, and combine 4ms. A single isolated forward chunk has a 16ms serial dependency chain. Now suppose a backward chunk from another micro-batch has 9ms of compute that can occupy part of the dispatch/combine intervals.

Overlap can shorten elapsed time, but not below every dependency bound. The experts still wait for their dispatch, and the next stage still waits for the combined result. The scheduler finds concurrency across different chunks rather than making one token's dependent operations simultaneous.

This toy example is not a reproduction of DualPipe's exact schedule. Its purpose is to distinguish legitimate cross-chunk overlap from an impossible claim that dependent dispatch, expert computation, and combine all execute at once for the same data.

What the report establishes

The report presents the schedule, resource comparison, custom kernels, and successful large-scale use. That is meaningful evidence of a workable integrated design. It does not publish enough timing data to independently reproduce a universal percentage speedup relative to every baseline.

To evaluate a port, compare end-to-end step time and peak memory under the same model, batch, numerical policy, and hardware. Standalone communication bandwidth is insufficient: the desired result is a shorter critical path while preserving training correctness and stability.

Source notes and further reading

DualPipe description, figures, and comparison table: main.tex, lines 478–538, §3.2.1. The 6.25% activation-measure increase is derived from the reported formula at PP16. Toy timing examples and overlap conditions are explanatory systems analysis.

Continue to Chapter 15: Communication.

15. Moving Tokens to Experts

MoE routing turns a regular tensor of token representations into groups of rows assigned to different experts. Those groups can reside on different GPUs and nodes. Efficient dispatch and combine are therefore essential to V3's conditional-computation advantage.

The report's communication design is tied to its H800 topology: fast NVLink/NVSwitch inside an eight-GPU node and slower InfiniBand across nodes. It uses locality constraints and specialized kernels to exploit that hierarchy while preserving the selected expert computation.

What dispatch must do

For each token, the router produces selected expert IDs and weights. Dispatch must send its hidden representation to the devices that host those experts. A token selecting several remote experts may need multiple copies or a representation that can be forwarded to several destinations.

The receiving side groups rows by expert so the expert's weight matrices can process a useful batch. Metadata identifies the original token and mixture weight, enabling the outputs to be restored and combined later.

An all-to-all pattern arises because each source rank can send different amounts to many destination ranks. It is irregular: message sizes depend on the router rather than a fixed uniform split.

What combine must do

Each expert produces an output for its assigned token rows. Combine returns those contributions and adds them according to the token's mixture weights, plus the shared-expert and residual paths as required by the model.

Correct combine must preserve token identity. A fast transfer with incorrect row ordering would corrupt the model even if every expert GEMM were numerically accurate.

Backward propagation requires related communication for gradients. The report's low-precision recipe uses FP8 for some dispatch-side activations or gradients but keeps combine in BF16. Thus dispatch and combine do not necessarily carry the same dtype or have identical cost.

The two network domains

The paper quotes 160GB/s for NVLink and 50GB/s for IB in the relevant communication design, a ratio of about 3.2. These are the report's topology-specific figures, not a general specification for every possible aggregate H800 link configuration.

Cross-node traffic is the more constrained resource. If a token selected eight experts spread over eight nodes, sending a full hidden representation independently to all eight would create substantial IB fan-out.

V3 restricts training routing to at most four target nodes. Once a token reaches a target node, faster NVLink forwarding distributes it to the local GPUs hosting selected experts.

Same-index cross-node transfer

The report describes first transmitting a token over IB to GPUs with the same in-node index on its target nodes. Those GPUs then forward it over NVLink to the exact local expert owners.

This routing structure helps organize cross-node transport and local fan-out. The kernel endeavors to forward arriving data promptly so later arrivals do not block earlier tokens' local delivery.

IB transfer and NVLink forwarding are overlapped. The intended result is that the slower link carries the necessary cross-node copies while the faster domain handles additional local destinations without adding the same amount of exposed delay.

Why the bandwidth ratio appears in the expert count

The report argues that each token can select an average of about 3.2 experts per target node without additional NVLink overhead under its overlap model. With at most four nodes, this gives approximately $4\times3.2=12.8$, described as a potential maximum of 13 routed experts at similar communication cost.

Actual V3 selects eight routed experts, not thirteen. The larger number is a systems capacity argument under the stated bandwidth assumptions. It is not a disclosed alternate model setting or a promise that increasing top-k to thirteen would preserve end-to-end cost.

More selected experts would still add expert computation, output combination, and perhaps load imbalance. “Same communication cost” is a narrower claim than “same training cost.”

A simple traffic estimate

A V3 hidden vector contains 7,168 elements. In FP8 its principal payload is about 7,168 bytes before scales and metadata; in BF16 it is about 14,336 bytes.

If a token's representation is sent once to each of four target nodes, the simple cross-node payload upper estimate is $4\times7168=28,672$ bytes in FP8. Sending eight independent full copies would be twice that.

This is only a teaching estimate. Some destinations are local, actual nodes selected can be fewer than four, scaling factors and indices add bytes, and transport buffers or forwarding layouts affect traffic. It should not be treated as a measured average for the training run.

Warp specialization

A GPU executes groups of threads called warps. Warp specialization assigns different warps to different tasks rather than making all threads follow one undifferentiated kernel path.

V3's dispatch kernel allocates warps for IB sending, IB-to-NVLink forwarding, and NVLink receiving. Combine uses warps for NVLink sending, forwarding plus accumulation into IB traffic, and IB receiving plus accumulation.

The report partitions twenty SMs into ten communication channels and dynamically adjusts warp allocations to workload. This supports progress across multiple transfer stages and helps avoid underutilizing one domain while another waits.

The number twenty is a reported tuning choice on the described hardware. It is not a universal optimal communication allocation for other GPUs, batch sizes, or topologies.

Why communication consumes compute resources

Streaming multiprocessors execute the instructions that move, organize, and reduce the data. While an SM handles communication, its Tensor Cores may be underused. Those SMs are also unavailable for other work during their allocated role.

The report notes twenty of the H800's 132 SMs used for communication in this design. The remaining compute capacity and interference must be considered alongside raw network bandwidth.

This is why overlap is not free. Communication can be hidden in elapsed time while still occupying resources. A poorly designed communication kernel can pollute cache, consume memory bandwidth, and slow the concurrent GEMMs enough to erase much of the gain.

Reducing interference

V3 uses customized PTX instructions and tunes communication chunk sizes to reduce L2 cache use and interference with compute kernels. Chunking controls how much data moves at once and how promptly downstream stages can begin.

Very large chunks can improve bulk efficiency but delay overlap. Very small chunks can increase instruction and coordination overhead. The useful point depends on bandwidth, message distribution, and the concurrent computation.

The report describes the strategy but does not release every exact production kernel or autotuning result in the TeX source. A port should profile L2 behavior, SM occupancy, sustained link utilization, and total step time rather than assuming a matching chunk size.

Why decode chooses another path

Prefill and training can use larger batches and hierarchical forwarding to favor throughput. Decode has small per-expert batches and stricter latency requirements.

The report's decode deployment instead uses direct point-to-point IB transfers with GPU-initiated communication support. Avoiding additional forwarding stages can help latency when messages are small and the deployment spreads experts more widely.

Thus the topology-aware training path is not the one mandatory communication algorithm for every V3 execution mode. Chapter 21 describes how serving chooses between these regimes.

Hardware suggestions follow from the bottleneck

The authors suggest offloading forwarding, buffer movement, reductions, and fine-grained layout management to communication hardware, with a unified interface across scale-up and scale-out networks.

The motivation is to stop spending scarce compute SM resources on transport orchestration. Such hardware could expose operations like multicast or reduce without forcing the main compute units to manage every detail.

These are proposals based on the reported implementation experience. They are not features that the V3 run already had. The report also proposes compute-side improvements for FP8, discussed in the next chapters.

Source notes and further reading

Topology, bandwidth figures, forwarding, twenty-SM allocation, and kernel tuning: main.tex, lines 540–558, §3.2.2. Hardware proposals: lines 632–658, §3.5.1. Decode communication: lines 611–629. Payload arithmetic is derived and excludes metadata and runtime overhead.

Continue to Chapter 16: FP8 numerics.

16. Understanding FP8 Numbers

FP8 uses eight bits to represent a floating-point value. Its small size can increase matrix throughput and reduce memory traffic, but it provides much less precision and range than BF16 or FP32. Successful training therefore depends on how values are scaled, grouped, multiplied, and accumulated.

V3 does not put every operation into FP8. It uses a mixed-precision design, with fine-grained scales and high-precision accumulation. This chapter explains the numerical principles; Chapter 17 follows the actual forward and backward pipeline.

Exponent and mantissa

A simplified normalized binary floating-point number has the form

where $s$ is a sign bit, $m$ is the fractional significand, and $e$ is an exponent. More mantissa bits distinguish nearby values more finely. More exponent bits allow a wider range of magnitudes.

E4M3 allocates four exponent bits and three mantissa bits, plus sign. E5M2 allocates five exponent bits and two mantissa bits. Exact subnormal, saturation, infinity, and NaN conventions depend on the format variant; the inspected released weights use the E4M3 finite-number convention identified by the official documentation.

V3 uses E4M3 for its principal FP8 tensors rather than E4M3 forward and E5M2 backward. The authors attribute the feasibility of this choice to small-group scaling, which compensates for the narrower dynamic range and lets the additional mantissa bit improve precision.

BF16 and FP32 play other roles

BF16 has a wide exponent range comparable to FP32 but fewer significand bits. It is useful for representing large and small values while using two bytes. FP32 provides more precise accumulation and optimizer state where repeated small updates matter.

The storage dtype of a tensor and the accumulation dtype of an operation are separate. A GEMM can multiply FP8 inputs and produce BF16 outputs while accumulating intermediate results at higher precision.

Similarly, using FP8 weights for a forward multiplication does not imply the optimizer's master weights are FP8. V3 retains FP32 master weights and accumulation gradients in its reported training recipe.

Scaling a group

To quantize a group $x$, choose a scale $s$ so $x/s$ fits the representable FP8 range. A conceptual operation is

The reconstructed $\widehat x$ is usually not exactly $x$. Rounding loses detail, very small values can underflow, and values beyond range can overflow or saturate depending on the scheme.

Some implementations store the reciprocal scale, so their formula is written with multiplication before quantization and an inverse factor afterward. The naming of scale versus scale_inv must be checked rather than inferred.

The official weight documentation says dequantization multiplies the FP8 weight block by its stored weight_scale_inv. That field is a reconstruction scale under the documentation's convention.

Why outliers make one scale inadequate

Suppose an invented tensor contains thousands of values near one and one value near one thousand. A single scale chosen from the maximum absolute value must accommodate the outlier. The ordinary values then occupy a much smaller part of the representable range and lose resolution.

A scale cannot simultaneously make all magnitudes large and preserve the largest without overflow. The problem is relative dynamic range inside the group.

Smaller groups isolate outliers so they affect fewer values. This does not eliminate quantization error, but it prevents one extreme feature from setting the scale for an enormous tensor.

V3's grouping

For activations, V3 groups one token by 128 channels, a 1×128 tile. For weights, it groups 128 input channels by 128 output channels, a 128×128 block.

The activation scheme prevents one token's outlier from setting another token's scale across a large block. The weight scheme amortizes scale metadata over many values while matching matrix-tile structure.

Using a hypothetical four-byte scale, one scale per 128 activation bytes adds about 3.125% metadata; one per 16,384 weight bytes adds about 0.0244%. These are derived illustrative overheads, not claims that every activation scale in the training system is stored as one FP32 value in this exact layout.

More granular scaling improves numerical control but increases scale computation, metadata handling, and kernel complexity. The optimal granularity is therefore an engineering tradeoff.

Scales along the GEMM inner dimension

A matrix multiplication sums products along an inner dimension $K$:

If quantization scales vary across $K$ blocks, the result is conceptually

A single global output scale cannot generally replace those different products of group scales. Each partial sum needs the appropriate dequantization factor before it contributes to the final result.

This is why V3's scheme requires more than invoking an ordinary per-tensor FP8 GEMM. The grouped-scale structure and accumulation path must be coordinated.

Multiplication error and accumulation error differ

Even if the quantized inputs are as accurate as possible, summing many products can lose information. A small term added to a large partial sum may round away. Positive and negative terms can cancel, making the final result small relative to the intermediate magnitudes.

A relative error metric can look especially large near zero because it divides by a small reference value. Numerical tests must therefore interpret maximum relative error alongside absolute error and the distribution of outputs.

The report observes limited accumulation precision in Hopper FP8 Tensor Core operations, retaining around fourteen bits in the relevant path. It reports nearly 2% maximum relative error in a preliminary random-matrix test with $K=4096$. This is a diagnostic experiment, not a statement that every V3 activation has 2% error.

Periodic high-precision promotion

V3 periodically transfers partial results to FP32 registers on CUDA cores and accumulates them there. The reported interval is $N_C=128$ inner-dimension elements, equivalent to four WGMMAs in the described implementation.

The scale factors for those groups can be applied during this promotion. This aligns dequantization with the high-precision partial-sum accumulation and reduces extra overhead.

For $K=4096$, a simple 128-element grouping gives 32 partial intervals. The exact instruction scheduling includes warpgroup concurrency: while one warpgroup performs promotion, another can execute matrix work. The goal is to improve accuracy without leaving Tensor Cores idle for every promotion.

This is still a mixed-precision compromise. It does not turn FP8 input multiplication into exact FP32 multiplication. It addresses the additional error from long low-precision accumulation chains.

Original illustration from DeepSeek-AI, report §3.3.2. Read the two panels together: group scaling and higher-precision accumulation are coordinated parts of the numerical recipe.

Online scales

Some low-precision systems estimate current scales using historical maximum values. V3 instead computes the maximum absolute value for the current tile or block and quantizes online.

This makes scale selection responsive to the current tensor. It avoids stale estimates but requires the scale calculation and cast to be executed efficiently.

The report proposes future hardware that fuses conversion with memory transfer to avoid reading BF16 data, writing FP8 data, and then reading those FP8 data again. This proposal follows from a real overhead in the software path; it is not a feature already assumed in the run.

Why transposition is not trivial

Forward activation tiles are grouped along one axis. Backward GEMMs may need the transposed orientation, changing which elements share a scale. V3 describes converting 1×128 grouping to 128×1 in relevant backward operations.

A tensor cannot always be transposed as raw FP8 bytes while retaining the original scale interpretation. The new groups may combine values with different previous scales. Dequantization and requantization can be required.

This is one reason a superficially simpler 128×128 activation scheme is tempting. But the report's block-wise gradient experiment diverges, showing that simpler layout can come at an unacceptable numerical cost.

What the precision experiments support

The paper compares FP8 and BF16 on approximately 16B and 230B MoE models trained for 1.33T and about 0.9T tokens respectively. It reports relative loss differences below 0.25% with the proposed scheme.

This supports the integrated numerical design at those tested scales and complements the successful full V3 run. It does not prove that every tensor is equally accurate, that all downstream tasks are unchanged, or that a different FP8 scheme would work.

The key lesson is that low precision is a recipe, not a checkbox. Grouping, accumulation, sensitive operations, and storage choices jointly determine whether the arithmetic remains compatible with learning.

Source notes and further reading

All V3 format, grouping, promotion, and online-scale details: content/fp8.tex, especially the sections on improved precision and mixed precision, §3.3. Hardware proposals: main.tex, lines 660–687. Comparative tests: lines 1458–1486, Appendix B. Floating-point equations and scale-overhead calculations are explanatory derivations.

Continue to Chapter 17: The FP8 pipeline.

17. The Actual Mixed-precision Pipeline

The phrase “FP8 training” can hide several different decisions. Which tensors are stored in FP8? Which multiplications accept FP8 inputs? Which sums accumulate at higher precision? Which operators remain BF16 or FP32? V3 answers these questions with a deliberately mixed design.

This chapter follows the linear layer through training and then connects its precision choices to attention, experts, communication, and the optimizer. It assumes only the idea that a matrix multiplication maps input rows into output rows.

One linear layer creates three GEMMs

Write a forward linear operation as $Y=XW$, where token rows are in $X$. Given an output gradient $dY$, backpropagation computes

These three matrix products are often named Fprop, Dgrad, and Wgrad. V3 performs the principal GEMMs in all three using FP8 inputs, with outputs in BF16 or FP32 and the controlled accumulation strategy described in Chapter 16.

Supporting FP8 Wgrad is especially valuable for memory because the saved forward activations needed by Wgrad can be stored in FP8. Supporting only FP8 Fprop would leave much of the backward compute and activation storage unchanged.

Original illustration from DeepSeek-AI, report §3.3.1. It depicts the linear operator, not every sensitive attention or normalization operation in the whole model.

A conceptual forward sequence

A linear input is produced at higher precision by preceding operations as appropriate. The system computes an online scale for each activation tile and casts the input into E4M3. Weight blocks are similarly represented with their scales.

The GEMM multiplies the quantized operands, periodically applies group scales to promoted partial sums, and emits a higher-precision output. Subsequent nonlinear, normalization, residual, or attention operations use their assigned formats.

The system may save a compact version of the activation for backward instead of the full higher-precision tensor. The exact saved representation depends on whether another operator needs that same activation with greater fidelity.

This is a data-flow description, not a claim that each conceptual step is a separate kernel. Good implementations fuse operations where possible to reduce launches and memory round trips.

High-precision exceptions

The report retains original higher precision for the embedding module, output head, MoE gating, normalization operators, and attention operators. These choices reflect sensitivity or low cost relative to the main GEMMs.

The labels describe the report's recipe. They are not a guarantee that every available inference engine uses the same formats.

Router precision has discrete consequences

Top-k routing has boundaries: a small score change can swap the selected expert. Quantizing a router carelessly can therefore change both the output path and the workload distribution.

Keeping gating in higher precision reduces one source of such perturbations. It also costs relatively little compared with the enormous expert matrices.

This illustrates why a precision policy should depend on an operator's role rather than only its FLOP count. A cheap operation can have a large structural consequence if its result determines subsequent control flow.

Attention-related saved activations

The input to the linear operation after attention can also be required by attention's backward pass. That backward path is sensitive to precision.

The report uses a customized E5M6 representation for these particular saved activations. It also discusses regrouping from 1×128 to 128×1 in backward and using power-of-two scales to avoid extra quantization error.

E5M6 is an exception to a simplistic “everything saved is eight bits” story. The report does not provide enough layout detail to infer an exact byte count for every packed E5M6 tensor. A memory model should not assume it has the same footprint as E4M3.

SwiGLU recomputation

Within MoE, the report caches SwiGLU inputs in FP8 and recomputes the SwiGLU output during backward. This avoids persistently saving an additional intermediate tensor.

The recomputation uses extra arithmetic but saves memory and bandwidth. Its desirability depends on the relative cost of recomputing the elementwise operation versus retaining its output throughout the activation lifetime.

Because the expert matrix operations dominate much of the computation, recomputing a comparatively cheap nonlinear intermediate can be favorable. The paper reports this as part of its successful memory design, not as a universal rule for every model shape.

Backward grouping and gradient sensitivity

The Dgrad path propagates information into earlier layers. Numerical error there can pass through many subsequent backward operations. Quantizing activation gradients with a group that combines very different token magnitudes can lose important small gradients.

The report's test using block-wise grouping for all Dgrad-related tensors causes divergence in an approximately 16B MoE run after around 300B tokens. The authors hypothesize that token-correlated gradient outliers explain the sensitivity.

The observed divergence is evidence; the proposed outlier mechanism is an explanation. Distinguishing them matters. A different block-wise scheme with other controls is not proven impossible by one failed experiment, but replacing V3's tile-wise design with a superficially simpler layout would require careful validation.

Optimizer state stays mixed

The optimizer stores first and second moments in BF16 instead of FP32. Master weights and accumulated gradients remain FP32.

A master weight records the optimizer's high-precision parameter state. A lower-precision compute copy can be generated for matrix operations. If tiny updates were accumulated only into FP8, many could disappear through rounding before they ever changed the compute weights.

BF16 moments reduce storage relative to FP32 moments. Their effect is reported as no observable degradation in the authors' investigation, but the report does not give a complete independent ablation for every optimizer-state choice at full V3 scale.

FP8 dispatch, BF16 combine

Before MoE up-projections, V3 quantizes activations to FP8 and dispatches them in a representation compatible with the experts' FP8 forward GEMM. It applies a similar strategy to a backward activation-gradient path.

This is efficient because communication and computation agree on the representation. Sending BF16 and immediately converting to FP8 on the destination would move more bytes than necessary.

Combine remains BF16 for both forward and backward in the report. It sums contributions from multiple experts, so preserving the combination's numerical fidelity is important. Low-precision transport is therefore selective, not a blanket reduction of every network payload to one byte per value.

Why theoretical doubling is not end-to-end doubling

The paper says FP8 principal GEMMs theoretically double compute speed relative to BF16 on the relevant hardware. End-to-end training also contains higher-precision operations, quantization, promotion, memory movement, communication, and scheduling overhead.

If half of a hypothetical workload were perfectly accelerated by two and the other half unchanged, total speedup would be $1/(0.5+0.5/2)=1.33$, not two. This is an Amdahl-style illustration, not a measured breakdown of V3.

FP8 also reduces saved activations and dispatch bytes, which can create benefits beyond the accelerated arithmetic fraction. Conversely, regrouping and conversion can add overhead. The integrated result must be measured.

What the released FP8 checkpoint means

The official repository distributes FP8 weights with block scales and provides a conversion route to BF16. These stored weights describe an inference artifact. They do not contain all FP32 master states and optimizer moments needed to resume the original training exactly.

Dequantizing FP8 weights into BF16 increases storage and changes the representation used by the inference GEMM. It does not recover precision that was already lost during quantization, nor does it create the historical training master weights.

The reference inference code demonstrates handling of block-scaled weights. It should not be presented as the complete training framework or proof of every production mixed-precision path.

Source notes and further reading

Precision assignments and storage/communication details: content/fp8.tex, §3.3.1–3.3.3. Block-wise Dgrad failure: main.tex, lines 1476–1486, Appendix B. Released scale handling: reference/_README_WEIGHTS.md. Linear backward equations and the hypothetical Amdahl calculation are background derivations.

Continue to Chapter 18: Memory and stability.

18. Memory Management and Reliability

Training memory includes much more than checkpoint weights. V3 must retain parameter state, gradients, optimizer moments, and enough forward information to compute backward gradients. Its pipeline schedule also keeps several micro-batches in flight.

The report reduces these costs through several coordinated techniques. Their importance is architectural: by making memory fit without tensor parallelism, they remove a frequent communication cost from the training plan.

A memory inventory

A useful conceptual inventory is

The categories can overlap in fused or sharded implementations, so this is a planning decomposition rather than an exact sum of separately allocated buffers.

Parameters include compute copies and master copies. Gradients can include accumulation buffers. Optimizer state includes moments. Activations have lifetimes determined by the pipeline schedule and recomputation policy. Runtime includes communication buffers, workspace, temporary casts, and allocator overhead.

Memory peaks when enough of these are live simultaneously, not necessarily at the point with the largest single matrix operation.

Why inference weight bytes are misleading for training

As an illustrative unsharded count, FP32 master weights plus FP32 gradients plus two BF16 moment arrays use $4+4+2+2=12$ bytes per parameter before the compute copy. Two FP32 moments would instead give sixteen bytes before that copy.

At 671B parameters, these illustrative logical arrays would total approximately 8.05TB or 10.74TB respectively. Actual V3 distributes and shards them; no individual GPU stores those totals.

The calculation shows the scale of the obligation and why even a two-byte saving per moment matters. It is not a complete per-GPU memory estimate, because PP, EP, DP sharding, DualPipe copies, and parameter-specific precision all affect placement.

Saved activations and their lifetime

Backward weight gradients need forward inputs. Nonlinear backward computations may need inputs or outputs. If all such tensors are retained at full precision for every layer and micro-batch, activation memory can become enormous.

An activation's cost depends on its shape and how long it remains live. A wide attention up-projection can be much larger than the compressed latent from which it was constructed. A pipeline schedule retaining many micro-batches multiplies these costs.

This makes lifetime analysis as important as dtype. Saving a tensor at FP8 helps; avoiding its persistent storage entirely can help more.

Recomputation

V3 recomputes RMSNorm operations and MLA up-projections during backward instead of retaining all their outputs. It also caches SwiGLU inputs and recomputes its output.

Recomputation trades arithmetic for memory. If the saved tensor is large and the operation is relatively inexpensive, the trade can be favorable. Recomputing the MLA up-projection allows the system to keep compact latents rather than all expanded per-head activations.

Recomputation must reproduce the needed forward values consistently enough for correct gradients. Random operations, numerical formats, and parameter versions require care. V3's attention dropout in the inspected release configuration is zero, but that configuration alone is not a complete account of every historical training operator.

Why normalization is a good candidate

RMSNorm uses a reduction, a scale, and elementwise operations. Its output can be large because it has one value per hidden component per token, even though its computation is modest compared with major matrix products.

Retaining the input and recomputing normalized output can therefore save a full activation tensor at relatively low extra arithmetic cost. The report says these recomputation choices incur minor overhead.

That conclusion is workload-specific. If an operation is expensive or its input is equally costly to retain, recomputation may offer less benefit. A system should measure the resulting peak memory and step time rather than applying the idea indiscriminately.

Optimizer sharding and precision

ZeRO-1 shards optimizer state across data-parallel ranks. BF16 first and second moments reduce each logical moment array's storage. PP and EP divide other parameter ownership.

These techniques address different axes: sharding changes which device stores a value; lower precision changes how many bytes the value requires; recomputation changes whether a value must remain stored at all.

They can be combined, but their savings should not be multiplied blindly. A tensor already omitted by recomputation cannot also contribute a separate persistent FP8 storage saving. A tensor replicated for schedule reasons can offset some of the sharding benefit. Accurate accounting follows actual allocations and lifetimes.

CPU exponential moving average

V3 maintains an exponential moving average of model parameters for early estimation of performance after learning-rate decay. A conceptual EMA update is

The report stores these EMA parameters in CPU memory and updates them asynchronously after each step. This avoids keeping another full EMA copy in GPU memory.

The authors describe negligible added time and GPU-memory overhead in their system. CPU memory, host-device transfer, and asynchronous bookkeeping still exist. “No GPU overhead” should not be expanded into “no resource requirement anywhere.”

The paper does not state enough to reconstruct the exact EMA coefficient, host placement, transfer schedule, or final checkpoint-selection policy. EMA here is a reported monitoring tool, not proof that the released checkpoint is necessarily an EMA-averaged one.

Physical MTP sharing

MTP shares the embedding module and output head with the main model. Under DualPipe, shallow and deep layers are placed on the same PP rank so these parameters and their gradients can be physically shared.

Logical sharing in equations is insufficient if a distributed implementation unnecessarily duplicates the arrays or maintains inconsistent gradient buffers. Placement makes the mathematical sharing produce an actual memory saving.

This is a recurring theme in the report: a good mathematical design needs a matching implementation to realize its resource benefit.

Stability has several layers

Numerical stability concerns finite values, bounded errors, and gradient propagation. Optimization stability concerns whether learning proceeds without destructive loss spikes. Operational reliability concerns device faults, communication failures, checkpointing, and restart behavior.

The paper's claim of no irrecoverable loss spikes or rollbacks addresses optimization history under its integrated recipe. The FP8 comparisons and block-wise failure study address numerical choices. The report does not provide a complete operational incident record.

It is therefore inappropriate to infer that no GPU failed, no job restarted, or every step had identical timing. Those are separate operational claims requiring separate evidence.

How smaller precision studies help

The successful FP8/BF16 comparisons at approximately 16B and 230B scales test whether the scheme tracks a higher-precision baseline over long runs. The failed block-wise Dgrad experiment identifies a concrete dangerous alternative.

The full V3 run adds evidence that the final combination scales to 671B. It does not replace every possible controlled ablation. The contributions of clipping, moment precision, recomputation, grouping, and accumulation are entangled in the integrated system.

For reproduction, the prudent order is to validate correctness on small deterministic examples, compare loss trajectories at modest scale, then stress the intended distributed and sequence-length regime. This is an engineering inference, not a procedure fully specified by the report.

Memory savings can alter the whole design

If memory pressure forces tensor parallelism, each layer can acquire extra synchronization. If memory fits without it, the system can use PP and EP with fewer dense-layer collectives.

Thus saving activation bytes can indirectly save communication time. The benefit is larger than the size of the omitted buffer alone. V3's report explicitly connects its memory work to avoiding costly TP during training.

This interaction explains why an apparently minor choice, such as recomputing normalization, belongs in the core story of model efficiency rather than in a footnote about implementation housekeeping.

Source notes and further reading

Recomputation, CPU EMA, and MTP sharing: main.tex, lines 560–577, §3.2.3. Optimizer/activation storage: content/fp8.tex, §3.3.3. Stability: introduction. Numerical comparisons: Appendix B. Memory inventories, EMA equation, and reliability distinctions are explanatory analysis.

Continue to Chapter 19: SFT and distillation.

19. Instruction Tuning and Reasoning Distillation

A base language model predicts continuations. An assistant must also recognize instructions, choose useful response forms, answer directly, and handle preferences. Supervised fine-tuning, SFT, teaches these behaviors from curated examples.

V3's report describes 1.5M instruction-tuning instances across multiple domains, with distinct generation methods for reasoning and non-reasoning data. Its reasoning pipeline transfers patterns from an internal DeepSeek-R1 model through domain-specialized teachers. This is a data-generation process, not a change that inserts R1 as a component inside V3 at inference.

What supervised fine-tuning optimizes

An SFT example contains a prompt and a desired response. A conventional objective rewards the model for producing the response tokens conditioned on the prompt and earlier response tokens:

The exact loss masking and normalization are implementation choices. The report does not disclose every one of them, so the equation is a standard explanatory form rather than a complete V3 loss specification.

SFT changes the distribution of text the model is rewarded for generating. It can teach formatting, task procedures, and answer styles while building on capabilities learned during pre-training.

Why not simply copy all R1 outputs

The authors report that R1-generated reasoning data are accurate but can be excessively long, poorly formatted, or prone to overthinking. The aim is to retain useful reflection and verification while producing clearer and more concise examples for V3.

More visible reasoning is not automatically better. It consumes output tokens and latency, can repeat work, and can make simple answers harder to use. Yet removing too much reasoning can lose intermediate checks that improve difficult-task accuracy.

The pipeline therefore optimizes a tradeoff rather than maximizing response length. This is central to distinguishing V3's post-trained behavior from a separate model optimized for long reasoning traces.

Domain teachers versus MoE experts

The report calls the data-generating domain models “expert models.” These are separate post-trained models specializing in domains such as mathematics, coding, and general reasoning.

They are different from the small MoE experts inside each V3 layer. A domain teacher can generate a complete natural-language solution. A MoE expert is a feed-forward transformation of one hidden vector.

Confusing these meanings can create a false picture in which the router selects an R1-like math assistant inside V3. The actual distillation process creates training examples that influence V3's own parameters.

The reported teacher-building pipeline

For each domain problem, the authors create two kinds of teacher SFT examples. One pairs the problem with an original response. The other pairs a system prompt and the problem with an R1-generated response.

The system prompt encourages reflection and verification patterns. The domain model is trained on this mixed supervision, then reinforced with high-temperature sampling so it can combine useful patterns from both response sources even without the explicit prompting scaffold.

After hundreds of RL steps, the resulting teacher is used to generate candidates. Rejection sampling selects high-quality examples for the final V3 SFT set.

The report outlines the sequence but does not disclose every domain model's size, all prompts, exact RL rewards, candidate counts, or selection thresholds. These are important missing ingredients for reproduction.

What high-temperature sampling contributes

Temperature changes the sharpness of a token distribution. For logits $z$, sampling at temperature $\tau$ uses approximately $\operatorname{softmax}(z/\tau)$.

Higher temperature broadens the set of sampled continuations, increasing diversity but often reducing average immediate quality. In teacher RL, this can expose alternative combinations of reasoning and style that reward feedback then selects.

The process is not guaranteed to discover every useful strategy. Its success depends on the initial model, prompt distribution, reward reliability, and sampling budget. The report states that it helps the teacher integrate R1 patterns, but does not supply a fully isolated temperature ablation.

Rejection sampling

Rejection sampling here means generating candidate responses and retaining those that meet quality criteria. The final student sees selected examples rather than every raw teacher output.

Suppose, in an invented example, a teacher generates four solutions to one problem. Two have incorrect final answers, one is correct but unnecessarily repetitive, and one is correct and clear. A filter can retain the last example for SFT.

This improves the training distribution without requiring the student to interact with the teacher during inference. It also spends generation and verification compute before training. Those costs should not be ignored when analyzing the complete development pipeline.

The selection process can introduce bias: problems that are easier to verify or that yield clear correct candidates may be overrepresented. Without the full filtering logs, the exact effect is unknown.

Non-reasoning data

For creative writing, role-play, and simple question answering, the report uses DeepSeek-V2.5-generated responses verified by human annotators. The data source and validation process differ from the reasoning-teacher pipeline.

Human review adds a way to assess quality that is difficult to encode as a deterministic reward. It also introduces annotation criteria and consistency questions. The paper does not provide the complete rubric or inter-annotator agreement statistics.

The 1.5M total counts instances, not tokens. Different response lengths can make the token budget much larger or smaller than a reader guesses from the instance count alone.

Final V3 SFT settings

V3-Base is fine-tuned for two epochs. The learning rate follows cosine decay from $5\times10^{-6}$ to $1\times10^{-6}$.

Multiple samples are packed into a sequence, but sample masking keeps them isolated. This allows efficient tensor utilization without letting one SFT example condition on another unrelated example's content.

This differs from the pre-training packing policy, which does not use cross-sample attention masking. An implementation copying the pre-training attention mask into SFT would miss a reported design choice.

Two epochs mean repeated passes over the curated set under the training procedure. They do not reveal the exact number of tokens, optimizer steps, or per-domain exposure without response-length and packing information.

What the distillation ablation shows

The report ablates the reasoning-data contribution on DeepSeek-V2.5, not full V3. On LiveCodeBench-CoT, pass@1 increases from 31.1 to 37.4 while average response length increases from 718 to 783. On MATH-500, performance rises from 74.6 to 83.2 while average length increases from 769 to 1,510.

The larger length change in math is especially informative. Accuracy and inference effort change together. The experiment supports the usefulness of the R1-derived data pipeline, but it does not prove the same gains at a fixed output-token budget or identify one particular reflection pattern as the cause.

The report says V3 settings were chosen to balance accuracy and efficiency. It does not disclose a complete optimization curve over every distillation setting.

Distillation does not copy a whole mind

The student learns from selected outputs. It can imitate useful procedures and may improve generalization, but it does not automatically inherit every teacher capability or internal representation.

Teacher mistakes can propagate. A response can be correct for the wrong reason, and a verifier checking only the final answer may retain flawed intermediate explanation. Conversely, a concise student can outperform a verbose teacher on some user-facing tasks if the training examples and inference behavior are better matched to the task.

The correct conclusion is that teacher-generated data can improve the student's behavior under tested conditions. Claims about identical reasoning processes would require different evidence.

What this chapter leaves to RL

SFT rewards matching selected demonstrations. Reinforcement learning instead rewards sampled behavior according to rule-based or model-based feedback. It can shift preferences among responses the model already knows how to produce and explore improvements where demonstrations are limited.

The next chapter explains V3's rewards and GRPO objective, including where the report is precise and where implementation settings remain unspecified.

Source notes and further reading

Dataset and teacher pipeline: main.tex, lines 947–973, §5.1. Distillation ablation: lines 1148–1179, §5.4.1. SFT and temperature equations are background; teacher-bias and compute-budget discussion is interpretation.

Continue to Chapter 20: Reinforcement learning.

20. Rewards, GRPO, and Self-feedback

Reinforcement learning changes a model by rewarding sampled outputs rather than only teaching it to imitate fixed examples. In V3, the feedback comes from deterministic rules when possible and learned reward models when necessary. Group Relative Policy Optimization, GRPO, compares several outputs for the same prompt and updates the policy without training a separate critic of comparable size.

This chapter derives the objective in plain language and examines what the rewards can and cannot establish.

The policy and sampled outputs

The language model is a policy $\pi_\theta$: a probability distribution over response tokens conditioned on the prompt and prior response. For a question $q$, an old policy produces a group of outputs $o_1,\ldots,o_G$.

A reward function assigns scalar scores $r_1,\ldots,r_G$. Training then increases the relative probability of better-scoring outputs and decreases that of worse-scoring ones, constrained to avoid excessively large updates.

The old policy is the source of the sampled training data. The current policy is being optimized. A reference policy supplies a separate regularization anchor. These three roles should not be collapsed into one model name.

Rule-based rewards

Some mathematical tasks have a deterministic final answer. A required output format, such as a boxed answer, permits extraction and checking. Coding tasks can be run against tests and compiler feedback.

These rewards are appealing because they do not depend entirely on another language model's subjective judgment. They are still limited by what is checked. Passing finite tests does not prove a program correct for all inputs. Correct final-answer extraction does not verify every intermediate explanation. A parser can be fooled or reject a valid differently formatted answer if poorly designed.

The report describes rule-based validation as more resistant to manipulation. That should be read as a comparative advantage, not a claim of immunity to reward hacking.

Model-based rewards

For free-form answers, a reward model can judge whether a response matches a reference answer. For creative or subjective tasks without a unique ground truth, it can assess quality using the prompt and response.

The reported reward model is trained from V3 SFT checkpoints. Its preference training data include a reasoning explanation leading to the reward, rather than only a final scalar or choice. The authors intend this to improve reliability and reduce reward hacking.

The presence of a judgment rationale does not guarantee correctness. A model can produce a plausible explanation for a biased or mistaken judgment. Validation requires comparing judgments with reliable human or task-specific criteria.

Group-relative advantages

GRPO normalizes rewards within a group:

An output above the group mean receives positive advantage; one below receives negative advantage. The score is relative to other samples for the same prompt, reducing the need for a critic to predict the absolute difficulty of every question.

In a toy group with rewards $[1,1,0,0]$, the mean is 0.5 and the population standard deviation is 0.5. The advantages are $[1,1,-1,-1]$ under that convention.

This normalization is not the same as using the raw reward as a gradient weight. It changes the scale across prompts and focuses the update on within-group differences.

The zero-variance case

If every sampled output receives the same reward, the displayed formula divides by zero. Implementations usually add a stabilizer or handle such groups explicitly, but the V3 report does not specify its exact rule here.

Conceptually, an all-equal group offers no relative preference signal. An all-correct group and an all-incorrect group both have this property, even though their absolute quality differs. Improving training on very easy or very hard prompts can therefore depend on sampling diversity, curriculum, reward granularity, or special handling.

This is a useful question for reproduction, not evidence that the actual training code crashed. The symbolic paper objective omits some implementation details.

The clipped policy objective

The report uses a likelihood ratio

and a clipped surrogate

For positive advantage, increasing probability helps until the clipping rule limits the rewarded increase. For negative advantage, the objective constrains the useful decrease differently through the same minimum expression. It does not merely clamp every gradient to a fixed magnitude.

Suppose $A_i=1$, $\epsilon=0.2$, and $\rho_i=1.5$. The two values are 1.5 and 1.2, so the surrogate takes 1.2. This avoids rewarding an arbitrarily large probability increase for that sampled output.

Clipping is a practical update constraint. It is not a proof that the entire distribution stays within a strict global distance bound after optimization.

Sequence notation versus token implementation

The V3 report writes the objective using output-level probabilities. An autoregressive response probability is a product of token probabilities, or a sum of token log probabilities. Long products can become extremely small.

Practical policy-gradient implementations often operate with token-level ratios, masks, and length normalization. The V3 text does not disclose every such choice. One should not invent an exact token-level implementation and claim it is fully specified by the displayed sequence-level equation.

Length normalization matters because long outputs contain more token decisions. Whether reward or advantage is assigned per token, per sequence, or with a length correction can change incentives toward verbosity.

Reference-policy regularization

The objective subtracts a KL-style penalty weighted by $\beta$, discouraging excessive movement from a reference policy. This helps preserve useful behavior and can limit exploitation of imperfections in the reward.

The report displays the sample expression

Since $r-\log r-1\geq0$ for positive $r$, it is a nonnegative pointwise quantity, zero when the probabilities agree. It is not the exact KL divergence for one sample. Under the appropriate expectation over the current policy, it can estimate the reverse-direction KL because the ratio term has expectation one.

The sampling distribution and implementation details matter for that interpretation. The paper's compact notation should not be turned into a claim of an exact full-distribution calculation on every update.

Why avoiding a critic matters

A critic estimates expected returns or values. In some large-model RL designs it can be comparable in size to the policy, adding training memory and computation.

GRPO obtains a baseline from sampled group scores instead. This removes that separate critic requirement, but the system still needs policy generation, reward evaluation, old-policy information, and a reference model or equivalent log-probability access.

A critic-free objective is therefore cheaper along one axis, not resource-free RL. The generation of multiple responses per prompt can be expensive, especially for long reasoning outputs.

Self-feedback and voting

For broader tasks, the report uses a constitutional-AI-style approach with V3's own voting judgments as feedback. Additional constitutional guidance specifies the desired direction of improvement.

On RewardBench, the table reports V3 at 87.0 average and majority voting over six judgments at 89.6. This supports the usefulness of aggregated judgments under that evaluation. It does not prove that the model can verify every answer it generates.

Voting can reduce independent noise. If all judgments share a misconception, their errors are correlated and voting may preserve it. Self-feedback needs external anchors and evaluation to avoid a loop that rewards the model's existing preferences rather than real correctness.

What the RL recipe leaves open

The report describes diverse prompt domains: code, math, writing, role-play, and question answering. It does not fully disclose group size, every reward coefficient, clipping and KL coefficients, rollout budget, exact number of final-model RL steps, prompt mixture, or zero-variance handling.

These omissions limit exact reproduction and causal attribution. They do not erase the reported gains, but they prevent an external reader from treating the displayed objective as a complete executable recipe.

The practical conclusion is that V3's final behavior reflects both supervised demonstrations and feedback-driven optimization. Benchmarks of that behavior must be read with the full post-training pipeline in mind.

Source notes and further reading

Reward models and GRPO equations: main.tex, lines 975–1016, §5.2. Generative reward evaluation: lines 1117–1143. Self-feedback: lines 1181–1191. Toy advantages, clipping example, KL-estimator explanation, and reproduction questions are derived analysis.

Continue to Chapter 21: Serving.

21. Serving Prefill and Decode

An inference server has two very different tasks. Prefill reads an existing prompt and builds attention state. Decode repeatedly generates one or more new tokens, reusing that state. V3 separates these stages so each can use a hardware arrangement suited to its workload.

The report describes H800 deployment units of 32 GPUs for prefill and 320 GPUs for decode. These are the authors' performance-oriented units for their service objectives, not mathematical minimums below which the model cannot run.

What prefill computes

A prompt's token IDs are all known. The model can process many positions in parallel while applying causal masking. It computes the hidden states needed to obtain the first next-token distribution and records reusable attention state for subsequent generation.

Prefill often supplies enough rows for efficient matrix multiplications. Long prompts also make attention pairwise work substantial. Time to first token includes queueing, input processing, prefill, scheduling, and transfer to the decode stage if disaggregated.

A model's advertised output tokens per second does not describe this whole delay. A service can have fast decode and slow first-token latency for long prompts.

What decode computes

After choosing a new token, the model processes its representation through every layer, updates the cache, and produces the following token distribution. Ordinary autoregression creates a dependency between steps.

A server batches one new token from each active sequence. Even with many requests, an MoE layer divides those rows among 256 routed experts, so individual expert batches can be small.

The report says decode batches per expert are usually within 256 tokens and describes memory access as the bottleneck in that regime. This is its workload observation, not a universal threshold applying to every engine or GPU.

Reported prefill deployment

The minimum unit in the report is four nodes, 32 GPUs. Attention uses TP4 with sequence parallelism and DP8. The MoE portion uses EP32.

Under the simple per-layer expert inventory, 256 routed experts across 32 GPUs give eight routed experts per GPU. The system adds 32 redundant expert copies, one additional copy per GPU, to improve load balance.

The dense MLPs in shallow layers use TP1 to avoid unnecessary tensor-parallel communication. This is a good example of choosing parallelism by operator rather than forcing every submodule into one uniform plan.

Why attention and experts use different groups

Attention follows the sequence and head structure. MoE follows expert ownership and token assignments. The best partitioning axes therefore differ.

TP4 attention divides its projection/attention work across a small group, while multiple data-parallel attention groups process different token batches. EP32 gives experts a larger pooled set of assigned tokens, improving matrix utilization.

Combining these views requires data-layout transitions. The report describes the grouping but does not supply all production tensor-layout and scheduling details. A schematic degree such as DP8 is not enough to reconstruct the exact transport plan.

Redundant experts preserve the learned function

If expert A is heavily used, its weights can be copied to another device. Tokens assigned to A can then be processed by either copy, with the same learned transformation subject to ordinary numerical execution differences.

This changes physical placement, not the semantic identity of the expert. It is different from rerouting tokens to unrelated experts solely to reduce load.

The report detects hot experts from online load statistics, updates the set periodically—for example every ten minutes—and rearranges experts within nodes to balance device work without increasing cross-node traffic. The interval is illustrative in the text rather than a universal fixed law.

Why training balance is insufficient

Training batches represent a broad corpus mixture. Serving traffic can concentrate on one domain, language, or time of day. A model whose experts are balanced over training can therefore have a highly skewed live workload.

Even if the total number of assignments is large, placement can put several hot experts on one GPU. That GPU then determines the layer's completion time while others wait.

Redundancy and placement respond to this drift. They are especially important because V3's batch-wise balancing permits stronger specialization. A learning choice that improves quality can require more sophisticated serving load management.

Reported decode deployment

Decode uses forty nodes, 320 GPUs. Attention uses TP4 with sequence parallelism and DP80; the MoE portion uses EP320.

For each MoE layer's placement, a GPU hosts one expert, with 64 GPU slots assigned to redundant and shared experts beyond the 256 routed-expert pool. “One expert per GPU” describes the layer's expert placement, not a claim that the device stores only one 44M-parameter function across the entire 61-layer model.

The shared expert is treated as an always-selected routed destination for transport purposes. Each token therefore dispatches to nine expert computations: eight conditionally selected routed experts plus the shared expert. The shared expert has not become optional or been absorbed into the top-eight learned selection.

Low-latency communication

Decode dispatch and combine use direct point-to-point IB transfers rather than the same hierarchical forwarding policy as prefill. The report also uses IBGDA to reduce latency through GPU-initiated networking support.

With each device's layer-level expert inventory narrow, weight movement and small-message latency are central. The report allocates a relatively small SM share to the MoE/communication path so attention can retain more compute resources.

This is a workload-specific optimization. A server on fewer devices, different links, or a quantized/offloaded weight layout may choose a different tradeoff.

Micro-batch overlap

Prefill processes two similarly sized computational micro-batches and overlaps attention/MoE computation from one with dispatch/combine from the other. Similar workload sizes reduce imbalance in the paired schedule.

For decode, the report says it is exploring paired micro-batches that overlap one attention path with the other's dispatch-plus-experts-plus-combine. It also describes dynamic redundancy as work under exploration.

These distinctions matter. A proposal or ongoing optimization should not be written as a fully deployed measured feature. The book treats the stated prefill overlap as implemented and the explicitly exploratory decode extensions as exploratory.

Dynamic redundancy

The report considers storing more experts on each GPU, such as sixteen, while activating only nine physical expert slots during a step. A runtime algorithm would choose a globally favorable placement/routing scheme before all-to-all.

Here “activating nine” refers to the physical workload scheduling example, not a redefinition of the model's eight-routed-plus-one-shared semantics. The extra stored copies create flexibility for dispatch.

Computing the schedule has a cost. Prefill's large matrix workload can amortize it; decode's latency-sensitive small batches need tighter integration with dispatch. The report does not provide a finished universal algorithm or complete measured results for this dynamic scheme.

Disaggregation has a transfer obligation

Separating prefill and decode allows specialized resource pools, but the decode stage needs the attention state corresponding to the prompt. That state must be transferred, made accessible, or otherwise reconstructed with correct ownership.

The report states stage separation without fully describing every KV-transfer and admission-control detail. MLA's compact state helps reduce the size of this obligation, but it does not eliminate it.

A production evaluation should include this transfer and queueing cost in time to first token. Measuring each pool in isolation can overstate the user-visible benefit.

Why smaller deployments remain possible

The official inference example describes running the model on fewer GPUs than the report's decode unit. Quantization, more experts per device, CPU offload, different batching, and weaker latency targets can alter feasibility.

The key distinction is between running the weights and serving them efficiently at the report's SLO and throughput. The paper explicitly identifies its relatively large recommended deployment unit as a limitation for smaller teams.

An operator should measure actual memory headroom, time to first token, inter-token latency, aggregate output throughput, and tail latency under real request mixtures. The 37B active count alone answers none of those deployment questions.

Source notes and further reading

Prefill/decode units, degrees, redundancy, communication, and exploration status: main.tex, lines 583–629, §3.4. Deployment limitations: lines 1208–1214, §6. Smaller demonstration deployment appears in the pinned official repository README. KV-transfer obligations and metric recommendations are systems interpretation.

Continue to Chapter 22: Speculative decoding.

22. Speculative Decoding with MTP

Generating tokens one at a time repeatedly invokes a very large model. Speculative decoding uses a cheaper proposal mechanism to draft future tokens, then checks those proposals with the target model. If enough proposals are accepted, the system emits more tokens per expensive target invocation.

V3's MTP module can supply the proposal. The report states an 85–90% acceptance rate for the second-token prediction across several generation topics and an observed 1.8× tokens-per-second improvement. The complete acceptance and serving implementation is not specified in the report, so this chapter explains the general correctness conditions separately from that measurement.

The two distributions

Let $p$ be the main model's next-token distribution at a particular prefix. Let $q$ be the draft distribution from a proposal mechanism, here potentially the MTP path.

The draft is cheaper because it does not require running the full main stack for every proposed continuation. It is also imperfect. Its distribution can differ from $p$, especially when the context or prior draft token is unfamiliar.

Simply emitting draft tokens would produce a different model distribution. Verification is the step that distinguishes speculative decoding from an unverified approximation.

How MTP supplies a second token

During training, MTP combines a main hidden state with the ground-truth next-token embedding and predicts the following token. During generation, the ground-truth next token is unavailable; a generated first token supplies that embedding instead.

A simplified sequence is: the main model supplies a first next-token decision, the MTP path proposes one additional token conditioned on it, and a target evaluation verifies the continuation. Efficient execution reuses hidden state and cache information rather than recomputing the entire prefix.

The exact batching and scheduling of these steps can vary. The report's training diagram is not a complete inference state machine.

Greedy verification

If the desired target behavior is greedy decoding, a simple conceptual rule accepts a draft token when it equals the target model's argmax at the same verified prefix.

At the first mismatch, the target's greedy token is used and the remaining draft suffix is discarded. This preserves greedy target decisions if the verification and cache update are correct.

Matching argmax tokens does not preserve stochastic sampling from the target distribution. A system using temperature or top-p sampling requires rules consistent with that sampling policy.

Distribution-preserving sampled verification

For ordinary exact speculative sampling, draw a proposed token $x$ from $q$ and accept it with probability

If rejected, sample a replacement from the normalized positive residual

These equations illustrate a standard target-distribution-preserving method. They are not claimed as a fully disclosed V3 production algorithm. They require probabilities under the same prefix and the actual draft distribution used to generate the candidate.

If sampling applies temperature, top-k, or top-p filtering, the verification distributions must reflect the intended transformed target and proposal policies. Checking untransformed logits while sampling transformed distributions can break the guarantee.

Why the correction works

The accepted probability mass for token $x$ is

The target mass not supplied by acceptance is $p(x)-\min(q(x),p(x))=\max(p(x)-q(x),0)$. Rejection occurs with total probability equal to the sum of this residual mass, so drawing from its normalized distribution restores the missing target mass.

Acceptance plus correction therefore produces $p$ under the stated conditions. This is a probability argument, not a claim that floating-point kernels or all practical approximate acceptance variants are perfectly exact.

A small distribution example

Consider two possible tokens, A and B. Let the target be $p=[0.6,0.4]$ and draft be $q=[0.8,0.2]$.

A is proposed with probability 0.8 and accepted with probability $0.6/0.8=0.75$, giving accepted mass 0.6. B is proposed with probability 0.2 and always accepted because $0.4/0.2>1$, giving accepted mass 0.2.

The remaining rejection mass is 0.2. The positive residual is entirely on B, so rejected proposals become B. Final probabilities are A=0.6 and B=0.4, exactly the target distribution in this idealized example.

Without correction, the output would be biased toward the draft's overpreferred token A.

Multiple draft positions

For several draft tokens, the target can evaluate their conditional distributions in one causally masked batch, provided each target position is conditioned on the proposed earlier prefix. Acceptance proceeds from left to right.

After a rejection, later proposals are no longer conditioned on the actual emitted prefix, so they must be discarded or recomputed. Some algorithms also emit an additional target token after all drafts are accepted. Expected tokens per pass then depend on the exact proposal length and bonus-token rule.

V3's depth-one MTP offers one extra future-token prediction. This book does not attribute a longer draft tree or specific bonus-token schedule to the report without evidence.

Acceptance is related to distribution overlap

For the standard sampled rule, expected acceptance of one proposal is

where total variation is half the summed absolute probability difference. A draft close to the target distribution has high acceptance.

This identity applies to that rule. Greedy-match acceptance measures something different. The report gives an acceptance percentage without enough implementation detail to infer that it is exactly this distribution-overlap statistic.

Acceptance can also vary with topic, temperature, prompt length, generation stage, and filtering policy. The reported 85–90% range is evidence from the authors' tested conditions, not a guarantee for every workload.

A useful simplified speed model

Assume a cycle emits one required token plus one optional accepted token, so expected output is $1+a$ tokens at acceptance probability $a$. Let ordinary single-token target time be $t_0$, and the combined draft/verification cycle time be $t_c$.

Then simplified speedup is

At $a=0.85$ to 0.90, the ideal gain would be 1.85–1.90 if the cycle cost equaled one ordinary step. A reported 1.8× is plausible when added work is modest, but acceptance alone does not prove the speedup.

This model intentionally caps each cycle at two outputs. An algorithm with a bonus target token or another draft schedule needs a different numerator. It is a teaching model, not a reverse-engineered timing breakdown of V3.

Cache correctness is part of correctness

Target verification can temporarily compute states for proposed tokens. If a proposal is rejected, the cache must correspond to the actual accepted prefix plus the replacement token, not to the discarded suffix.

MTP can have its own state requirements as well. Position counters, page allocation, attention masks, and request batching must all agree after acceptance or rejection. An apparently correct token acceptance rule can still produce wrong future behavior if stale draft cache entries remain visible.

This is why enabling MTP support requires more than loading its additional weights. The serving engine must integrate drafting, target checks, and state rollback or truncation.

How to read the 1.8× claim

The paper reports tokens per second, not a universal reduction of every latency component. Prompt prefill, queueing, cache transfer, and long output scheduling may be unaffected or affected differently.

A thorough evaluation would compare the same prompts, target sampling settings, concurrency, hardware, cache dtype, and output-length distribution, then measure output throughput and per-request latency. The report does not provide the full matrix of these conditions for its MTP speed figure.

The defensible conclusion is that V3's additional predictor is accurate enough to support a substantial measured decode improvement in the authors' system. The exact gain in another engine requires direct measurement.

Source notes and further reading

MTP inference use: main.tex, lines 453–455, §2.2. Acceptance and TPS claims: lines 1193–1199, §5.4.3. Acceptance/correction equations, total-variation identity, and speed model are standard derivations provided for explanation, not a disclosed complete V3 implementation.

Continue to Chapter 23: Base evidence and ablations.

23. Base-model Evidence and Ablations

A final benchmark table answers whether a model performs well under a protocol. An ablation asks a narrower causal question: what changes when one design choice changes and other factors are held sufficiently similar?

V3's report contains both kinds of evidence. This chapter examines the base-model comparisons and the smaller-scale experiments for MTP, balancing, and FP8. It separates the strength of the evidence from the authors' broad headline descriptions.

What a base model evaluation measures

A base checkpoint has not yet received the final assistant post-training. It can still solve tasks when prompted with examples or a suitable completion format. Such evaluation measures useful learned capabilities without the final instruction-tuning pipeline, although prompt engineering and decoding remain influential.

The report compares V3-Base with V2-Base, Qwen2.5 72B Base, and LLaMA-3.1 405B Base using the authors' internal evaluation framework and shared settings.

Shared settings improve comparability. They do not make the models equal in training data, tokenizer, total capacity, active arithmetic, or development budget. This is a model comparison, not a controlled experiment in which only one architecture choice differs.

Likelihood-based versus generation-based tasks

For several multiple-choice or continuation tasks, the report uses a perplexity-based evaluation. The model's probabilities determine which answer or continuation is preferred. It need not generate a free-form explanation.

For code, math, question answering, and other tasks, the model generates a response and a metric checks that response. This introduces answer-format, parsing, sampling, and output-length effects.

A likelihood-based accuracy and a generation-based pass@1 can both be useful, but they measure different behavior. A model can rank the correct option well while struggling to follow the format needed to emit it reliably.

Bits per byte

Token-level perplexity depends on the tokenizer. A model that uses fewer tokens for the same text can have a different per-token loss even when its text-level probability is comparable.

The report evaluates Pile-test using bits per byte, BPB:

Lower is better. Normalizing by raw bytes makes the metric more comparable across tokenizers than per-token perplexity. It still depends on the exact text, encoding, boundary handling, and evaluation procedure.

V3-Base scores 0.548 versus LLaMA-3.1 405B's 0.542 and Qwen2.5 72B's 0.638. The dense LLaMA model is better on this aggregate language-modeling metric, even though V3 leads many task benchmarks. A strong model profile need not dominate every measure.

Representative base results

These results support broad strength in knowledge, code, mathematics, and multilingual subject questions under the report's settings. The raw local table contains additional tasks and metrics.

Counterexamples are informative

V3-Base is not the best on every row. RACE-Middle is 67.1 versus LLaMA's 74.2; RACE-High is 51.3 versus 56.8. NaturalQuestions is 40.0 versus 41.5. PIQA is 84.7 versus 85.9. CMMLU is 88.8 versus Qwen's 89.5.

The paper's caption considers scores within 0.3 to be at the same level. That is a presentation convention, not a statistical confidence interval. It does not establish that every larger gap is significant or every smaller gap is noise.

The pattern is more useful than a claim of universal superiority: V3 improves strongly in several target domains but retains weaknesses and task-dependent behavior.

What the MTP experiment controls

The report trains paired models with the same data and base architecture, adding one MTP module to the treatment. The module is removed at evaluation, so the evaluated main-model architectures and inference costs match.

At 15.7B total parameters and 1.33T tokens, HumanEval rises 20.7→26.8 and GSM8K 25.4→31.4. At 228.7B total and 540B tokens, HumanEval rises 44.5→53.7 and MATH 38.6→39.8.

This is useful causal evidence that the added training objective/module can improve the learned main model. It does not control total training FLOPs, because the MTP treatment executes extra work. It also does not provide repeated independent seeds or full uncertainty intervals.

Why exact point gains should not be overgeneralized

The smaller MTP model gains 6.1 points on HumanEval; the larger gains 9.2. Those numbers are specific outcomes of those model/data regimes. They are not constants that can be added to V3's score or another architecture's baseline.

Some metrics decline: large MMLU 67.5→66.6, small NaturalQuestions 22.7→22.3. This supports “improves many tasks” rather than “uniformly improves performance.”

A stronger study of efficiency would compare equal-compute alternatives: MTP versus additional ordinary training tokens, a wider baseline, or another auxiliary objective. The report does not supply that complete frontier.

The balancing ablation

The balancing comparison uses 15.7B models trained on 1.33T tokens and 228.7B models trained on 578B tokens. The baselines use auxiliary losses, with sigmoid gating and selected-affinity normalization already in place. The treatment removes those auxiliary losses and uses the bias-based balancing strategy.

This is important: the comparison is not simply V2's old score function versus V3's new one. The authors align the main gating transformation in the baselines.

At the larger scale, HumanEval rises 40.2→46.3 and GSM8K 70.7→74.5, while MMLU decreases 68.3→67.2. Again, the broad positive pattern has exceptions.

The treatment in this ablation removes all auxiliary losses. Full V3 retains a tiny complementary sequence loss. The ablation's “auxiliary-loss-free” variant and the final recipe are therefore related but not exactly identical configurations.

The balancing-scope experiment changes the explanation

A separate comparison tests batch-wise auxiliary loss. At similar batch balance, it achieves the same validation loss as the bias method: 2.253 at 1B scale and 2.080 at 3B scale, outperforming sequence-wise loss at 2.258 and 2.085 respectively.

This is evidence that allowing imbalance within individual sequences can explain at least part of the gain. It weakens a simplistic causal story that the absence of balance-loss gradients alone must be responsible.

The domain expert-load plots provide a plausible mechanism: more flexible batch-wise constraints permit stronger domain specialization. They do not prove that every specializing expert improves every task or that the full V3 gain can be decomposed uniquely into this mechanism.

FP8 validation

The FP8 study compares approximately 16B and 230B MoE baselines with BF16 over 1.33T and around 0.9T tokens. The reported relative loss differences remain below 0.25% using the proposed quantization and accumulation design.

This tests numerical fidelity of the integrated scheme. It does not show that every conceivable eight-bit scheme works. A contrasting experiment using block-wise Dgrad quantization diverges at about 300B tokens in a 16B run.

The failed experiment adds specificity: preserving token-wise gradient range matters in the tested setup. Its proposed explanation, token-correlated outliers, remains a hypothesis rather than an exhaustive proof of the failure mechanism.

Original comparison from DeepSeek-AI, report Appendix B. Curves are smoothed using EMA coefficient 0.9. The axes show consumed billions of tokens and training loss; this is a numerical-training comparison rather than a downstream benchmark table.

Measurement uncertainty

For an accuracy estimate $\widehat p$ over $n$ independent examples, a rough standard error is $\sqrt{\widehat p(1-\widehat p)/n}$. At $p=0.9$ and $n=500$, it is about 1.34 percentage points.

This is an illustrative binomial calculation. Real benchmark comparisons may use paired examples, correlated questions, repeated sampling, and deterministic decoding, so the appropriate uncertainty analysis differs. It nevertheless shows why a 0.3-point bolding threshold is not itself a significance test.

Repeated sampling on one checkpoint measures decoding variability. Repeated independent training seeds measure training variability. They are different sources of uncertainty and should not be conflated.

What can be concluded

The base-model table establishes a strong capability profile under shared protocols. MTP and balancing ablations provide evidence for improvements in related smaller-scale settings. FP8 comparisons and the full run support the numerical and systems feasibility of the integrated recipe.

The report does not isolate every contribution at full scale, match all comparisons by total development cost, or publish a complete reproducible dataset and training system. A careful reader can recognize the significance of the results while keeping those boundaries explicit.

Source notes and further reading

Base evaluation setup: main.tex, lines 767–823, §4.4. Scores: tables/base_evaluation.tex. MTP: lines 825–866. Balancing: lines 868–932. FP8: Appendix B. Uncertainty formulas and causal critiques are analysis.

Continue to Chapter 24: Chat evidence.

24. What the Chat Evaluations Establish

The chat checkpoint combines pre-training, long-context adaptation, SFT, reasoning distillation, and RL. Its evaluation therefore measures the resulting assistant, not an isolated architecture change.

The report compares specific historical snapshots: Qwen2.5 72B Instruct, LLaMA-3.1 405B Instruct, Claude-3.5-Sonnet-1022, GPT-4o-0513, and earlier DeepSeek models. These identifiers matter. The results are not a current ranking of every later version of those families.

The shared output budget

The report allows up to 8,192 output tokens per benchmark response. For small datasets, it uses repeated evaluations with varied temperatures to improve robustness.

A fixed token cap makes one constraint common, but different tokenizers can encode different amounts of text in that many tokens. Models may also use very different average output lengths while remaining below the cap.

Equal maximum output length is therefore not equal test-time compute. A model generating a long solution spends more decode work than one answering briefly, and models differ in per-token cost.

Knowledge and instruction-following results

V3 is competitive across these tests but not uniformly best. Claude leads MMLU-Pro and GPQA-Diamond in this table. IFEval differences near the top are small, and the report does not provide full confidence intervals.

MMLU and GPQA are tests of answering their question distributions, not exhaustive inventories of knowledge. IFEval uses verifiable instruction constraints, which are useful but do not cover every conversational preference or ambiguous user intent.

Factuality is a separate weakness profile

On SimpleQA, V3 scores 24.9, compared with GPT-4o's 38.2 and Claude's 28.4. On Chinese SimpleQA, V3 scores 64.8, compared with GPT-4o's 59.3 and Claude's 51.3.

This contrast supports language- and dataset-dependent factual performance. It also shows why high multiple-choice knowledge scores should not be translated into a universal factual reliability claim.

A correct-answer rate of 24.9% on SimpleQA does not mean that 75.1% of every V3 answer in ordinary use is a hallucination. The dataset, scoring, question difficulty, possible abstention behavior, and prompt distribution differ. It is a warning against conflating benchmark knowledge and reliable open-ended factual recall, not a general-purpose error-rate estimate.

Code results need several categories

The table supports strong code generation and competition performance. Repository maintenance has a different pattern: Claude leads SWE-bench Verified and Aider-Edit, while V3 leads Aider-Polyglot in the reported setup.

A single coding score would hide this distinction. Writing a short function, solving a contest problem, applying a patch, and resolving an issue in a repository involve different context, tools, and constraints.

The harness is part of a coding result

HumanEval-Mul covers eight languages in the report: Python, Java, C++, C#, JavaScript, TypeScript, PHP, and Bash. LiveCodeBench uses problems collected from August through November 2024 and reports CoT and non-CoT configurations separately.

SWE-bench Verified is evaluated with the Agentless framework. Aider tasks use the diff format. These setup details determine how the model accesses code and expresses edits.

A resolved-issue score is therefore a model-plus-harness result. Changing repository navigation, test access, retry policy, context selection, or patch format can change performance even with identical weights.

The report's code results do not establish that V3 is universally the best coding agent in every workflow. They identify strengths under specific controlled formats and tool scaffolds.

Mathematics and inference effort

AIME and CNMO use temperature 0.7 and average sixteen runs. MATH-500 uses greedy decoding. Averaging pass@1 over repeated runs is not pass@16: it does not mean the benchmark grants success whenever any of sixteen attempts is correct.

V3's gains are substantial under these protocols. They can reflect better learned reasoning patterns, data coverage, and post-training. They do not isolate which component supplied how much of the gain.

The base table's MATH score of 61.6 and chat table's MATH-500 score of 90.2 use different evaluation scopes and protocols. Subtracting them to claim a 28.6-point causal post-training gain would be invalid.

Long context and Chinese tasks

V3 scores 48.7 on LongBench v2 and 73.3 on FRAMES, compared with GPT-4o's 48.1 and 80.5 respectively. This supports useful context-related capability but shows different relative strengths across tasks.

On C-Eval, V3 scores 86.5 versus Qwen's 86.1. On CLUEWSC, V3 scores 90.9 versus Qwen's 91.4. Along with the Chinese SimpleQA result, these indicate strong Chinese performance without universal dominance across every Chinese benchmark.

Multilingual coverage beyond English and Chinese is reported, but the evaluation suite is weighted toward those languages. A user deploying in another language should not infer equal reliability merely from the model's multilingual training description.

Open-ended preference evaluation

The report uses AlpacaEval 2.0 and Arena-Hard with GPT-4-Turbo-1106 as the judge under their original configurations. AlpacaEval uses length-controlled win rate.

These are judge-mediated preference scores, not direct percentages of objectively correct answers. Arena-Hard is evaluated against its baseline; an 85.5 score is not an 85.5% head-to-head win rate against every model in the table.

A source inconsistency worth preserving

The Arena-Hard table gives 85.5 for V3. The accompanying prose says “over 86%.” This book uses the table's explicit 85.5 and records the discrepancy rather than silently harmonizing it.

Likewise, AlpacaEval rises from 50.5 for V2.5 to 70.0 for V3, a difference of 19.5 percentage points. A prose description of roughly 20% improvement should not be treated as an exact relative percentage calculation.

Small editorial discrepancies do not invalidate the whole report. They do show why a careful reader should inspect raw tables rather than copying headline wording.

What length control does and does not do

Judges can prefer longer responses because they appear more complete. Length-controlled evaluation attempts to reduce that confounding effect.

It does not remove every stylistic bias, judge preference, or correlation with the model's training style. A response can be persuasive yet wrong, and a judge may share the evaluated model's blind spots.

Open-ended benchmarks add evidence about usefulness and presentation that strict-answer tests miss. They should complement correctness, execution, and human evaluation rather than replace them.

Generative reward judgment

RewardBench evaluates a model's ability to choose preferred responses. V3 scores 87.0 average; majority voting over six judgments reaches 89.6. Claude-Sonnet-1022 scores 88.7 and GPT-4o-0806 86.7 in that table.

The GPT snapshot differs from GPT-4o-0513 in the main chat table. Cross-table comparisons should preserve that distinction.

A high RewardBench score supports using the model as a judge in similar conditions. It does not imply infallible self-verification or prove that its own generated answers are equally reliable. Judging a pair and generating a correct solution are different tasks.

What remains uncertain

The report does not provide full raw outputs, complete uncertainty analyses, every prompt serialization, all benchmark contamination audits, or identical compute budgets across models. Closed-model training resources and architecture are also not disclosed.

These limitations affect how precisely one can rank models and attribute gains. They do not negate the consistent evidence of strong V3 performance across many domains.

For practical evaluation, choose real tasks, fix the model snapshot and harness, record prompt/output budgets, and measure quality alongside latency and cost. The paper's tables are an excellent starting hypothesis, not a substitute for that application-specific evidence.

Source notes and further reading

Protocols: main.tex, lines 1018–1042, §5.3.1. Standard scores: tables/chat_evaluation.tex. Open-ended table and prose: lines 1082–1115, §5.3.3. RewardBench: lines 1117–1143. Interpretation of metric scope and discrepancies is analysis grounded in those sources.

Continue to Chapter 25: Costs and limits.

25. Costs, Limitations, and Practical Use

V3's reported training cost is one of the paper's most discussed results. It is meaningful only when its boundary is preserved: the authors estimate the resource cost of the final reported training stages, not the complete cost of discovering the recipe or operating the resulting service.

This chapter reconstructs the arithmetic, distinguishes capacity from performance, and turns the paper's limitations into practical questions for model users and researchers.

The reported accounting

The multiplication is direct. The $2 rate is an assumption in the report, not a current market quote or a guarantee that another team can rent an equivalent cluster at that effective price.

The paper explicitly excludes prior architecture, algorithm, and data research and ablations. It does not separately itemize every upstream teacher-training and data-generation expense. One should preserve that stated boundary rather than either ignoring the figure or treating it as the whole organizational budget.

GPU hours versus elapsed days

At continuous use of all 2,048 GPUs, pre-training elapsed time implied by the resource total is

The total for all listed stages would correspond to approximately 56.72 full-cluster days if they all used that same continuously occupied cluster. The actual later-stage hardware arrangement and utilization need not match this simplifying assumption.

The paper says a trillion pre-training tokens requires 180K GPU hours, about 3.7 days on the full cluster. Multiplying 180K by 14.8 gives 2.664M, consistent with the table.

These derivations are checks of internal arithmetic. They do not independently verify the job logs or hardware billing.

Implied token throughput

Dividing the 14.8T pre-training tokens by 2.664M GPU hours gives approximately

Multiplying by 2,048 gives approximately 3.16M training tokens per second across a continuously occupied full cluster.

This is average processed-token throughput implied by the stated resource budget. It is not per-user generation speed, nor does it mean one GPU alone can train the complete model at that rate. The model's work is distributed across the cluster.

Training throughput includes many positions whose targets are known and processed in parallel. Decode throughput operates under sequential unknown-token dependencies, so the two numbers should not be compared directly.

A rough FLOP check and its limits

A common rough training estimate is $C\approx6ND$, where $N$ is a dense-equivalent active parameter count and $D$ is the token count. Substituting rounded $N=37$B and $D=14.8$T gives

Dividing by the reported GPU seconds gives roughly 343TFLOP/s per GPU as an implied effective arithmetic rate under this crude model.

This is not a measured model-FLOP utilization figure. The approximation ignores details of embedding lookups, output projection, attention pairwise work, MTP overhead, recomputation, and the exact active-count convention. Comparing it with a hardware peak without those adjustments would be misleading.

Its value is pedagogical: it shows the enormous total computation even when the dollar headline appears modest. Efficiency changes the coefficient and hardware utilization; it does not make the training run small.

Why capacity comparisons remain ambiguous

V3 stores 671B main-model parameters but activates about 37B per token. A dense 405B model uses much more of its stored capacity on each token, while a dense 72B model uses all of its major feed-forward matrices.

Comparing scores at equal active parameter count asks about conditional-computation efficiency. Comparing at equal total parameter count asks about storage capacity. Comparing at equal training FLOPs asks about learning efficiency. Comparing at equal deployment cost asks about a full systems workload.

These are different experiments. The report's main tables do not equalize all of them, and closed-model training budgets are not fully available. A broad claim that V3 is better under every cost normalization goes beyond the evidence.

Raw inference-weight memory

As a simple lower-level illustration, 671B values occupy about 1.342TB at two bytes per value or 671GB at one byte per value, before scales and mixed-precision exceptions. Four-bit storage would give roughly 335.5GB of raw values before its own overhead.

These are hypothetical raw-size calculations. They do not establish that a particular quantization preserves quality, that all tensors use the same format, or that runtime fits at exactly those totals. The downloadable artifact can include MTP weights, duplicated shared tensors, and scale metadata.

At one byte per value, dividing 671GB by 80GB gives about 8.39 devices' worth of nominal capacity. Nine such devices would be a bare aggregate-byte bound under unrealistic uniform storage assumptions, not a usable deployment plan. The system needs cache, workspace, buffers, replication, and headroom, and device capacities and layouts vary.

Why the serving recommendation is large

The report's decode unit uses 320 GPUs to meet its latency and throughput objectives with one expert slot per GPU per layer and redundant copies. This is far larger than a bare weight-storage lower bound.

The extra scale buys parallel expert access, narrow per-device weight loading, attention groups, load balancing, and aggregate request capacity. It is a performance deployment choice rather than proof that the model cannot run on fewer GPUs.

The authors identify this relatively large recommended unit as a burden for smaller teams. They also report more than twice V2's end-to-end generation speed in their deployment while noting further room for improvement.

What a small team can decide

A team choosing among hosted access, a shared inference cluster, and self-hosting should begin with its actual workload: prompt lengths, response lengths, concurrency, domain mix, and acceptable latency.

Self-hosting offers direct control over weights and execution but requires substantial memory and systems expertise. A hosted endpoint avoids that infrastructure obligation but introduces service-specific limits and model-snapshot behavior. The report does not provide current pricing or a universal total-cost comparison.

A practical pilot should measure correctness on real tasks, time to first token, inter-token latency, total throughput, and memory peaks. It should also record the exact checkpoint and inference settings so changes are interpretable.

Quantization is another experiment

Running a lower-bit variant can reduce storage and make deployment feasible on smaller hardware. It can also change model quality, routing, or attention behavior if sensitive tensors are quantized poorly.

The original FP8 training recipe demonstrates one carefully controlled low-precision method. It does not automatically validate every INT4, weight-only, or cache-quantization conversion offered by a serving engine.

Evaluate the converted model on the actual task distribution. A broad benchmark average can hide a regression in a specialized domain or a rare format constraint important to the application.

Openness and reproducibility

The report calls the models open-source and releases weights plus useful inference material. Exact reproduction still lacks the full corpus, complete training implementation, all post-training settings, and comprehensive operational traces.

It is useful to distinguish open weights, available inference code, and a fully reproducible training pipeline. These are different levels of access, each valuable for different work.

The pinned repository uses separate code and model-license documents. This book does not infer the current license of later variants from a historical V3 snapshot. If licensing determines a deployment decision, consult the exact artifact's terms rather than an informal family label.

Scientific limits beyond deployment

The paper does not prove that the architecture is optimal, that every reported gain comes from a unique mechanism, or that benchmark leadership transfers unchanged to every real task. Its evaluation profile includes factuality and engineering-task gaps.

Its stability claim covers the reported integrated run. Its MTP speed claim covers a tested serving setup. Its long-context evidence covers specified retrieval and benchmark tasks. Each is substantial but bounded.

The right response to these boundaries is targeted follow-up work: equal-compute ablations, real workload serving profiles, corpus audits, richer long-context tasks, and external evaluation of rewards. These questions follow from the paper's strengths as much as from its omissions.

The lasting engineering lesson

V3 demonstrates that model quality and efficiency can be improved together when the design accounts for learning, numerical formats, memory lifetimes, and network topology. An isolated feature copied without its supporting system may fail to reproduce the benefit.

For research, the useful question is which coordinated choices transfer to a new scale or hardware regime. For deployment, it is which parts of the capability profile matter enough to justify the storage and operational complexity.

Source notes and further reading

Cost table and exclusions: main.tex, lines 186–208, §1. Deployment limits and speed comparison: lines 1201–1228, §6. Serving units: §3.4. All throughput, FLOP, and raw-memory figures are derived under explicit assumptions; they are not new measurements or current price recommendations.

Continue to Chapter 26: Worked traces.

26. One Request and One Training Step

Mechanisms become easier to understand when followed through one concrete process. This chapter traces a user request through inference and a batch through training. The request and sizes are invented for explanation; the component designs follow the report.

The traces are conceptual rather than exact production logs. Where the report does not disclose orchestration details, the chapter identifies the obligation without inventing an implementation.

Trace A: a document-and-code request

Imagine a user supplies a technical document and a code snippet, then asks the model to explain a bug and suggest a corrected function. After chat serialization and tokenization, suppose the prompt contains 8,000 tokens.

The application may add role markers or other prompt formatting. The model receives the resulting token sequence, not an abstract structured request. Exact serialization therefore matters to the distribution it sees.

Step 1: token IDs and embeddings

Each ID selects a row from the embedding table, producing a 7,168-dimensional vector. The representation initially reflects token identity. Later layers make it sensitive to the surrounding document and code.

An 8,000-token prompt has known input positions. Prefill can process them in parallel under a causal mask. This is different from generating 8,000 unknown output tokens.

Step 2: the first attention layer

For each position, MLA projects the normalized representation into a 512-dimensional content KV latent and a 64-dimensional positional key. It also forms a 1,536-dimensional query latent and head-specific query components.

Every head scores eligible earlier positions using content and positional terms. Softmax weights combine value information, and the attention output projection maps the concatenated head outputs back to width 7,168. A residual addition updates the stream.

The efficient serving implementation retains the content latent and positional key for future decode. It may use a different transient prefill layout to execute the bulk attention efficiently.

Step 3: the first three feed-forward layers

The initial three layers use dense SwiGLU FFNs. Every position receives the full dense transformation at those layers.

This establishes that V3 is not an all-MoE stack. Conditional experts begin after the early dense layers. The rationale for this exact count is not separately ablated in the V3 report.

Step 4: an MoE layer

At the next layer's feed-forward stage, the router computes 256 affinities from each contextual hidden vector. Correction biases and group restrictions affect selection. Eight routed experts are chosen, and the shared expert is always used.

The selected set for a prose token can differ from the set for a code identifier. It can also differ at another layer for the same token. No explicit “choose the coding assistant” instruction is required inside the router.

The server groups assignments, dispatches token rows to expert owners, executes their matrix operations, and combines the outputs. Redundant copies can process hot experts without changing the intended expert identity.

Step 5: repeat through 61 layers

Attention and feed-forward updates repeat. Each layer has its own cached state and expert functions. Information from the bug description, code snippet, and document can be integrated through successive attention paths.

The existence of that path does not guarantee a correct diagnosis. The model's learned features and post-training behavior determine whether it uses the relevant details well.

Step 6: first output distribution

The final hidden state at the last prompt position is normalized and projected to vocabulary logits. The generation policy selects the first answer token using greedy or configured sampling behavior.

SFT and RL influence this distribution through the learned weights. They are not separate runtime judges automatically checking the answer. Unless the application explicitly calls tools or validators, no compiler or reward model runs merely because coding rewards were used in training.

Step 7: cache transfer and decode

In a disaggregated deployment, decode must obtain the correct prompt cache. The report does not spell out every transfer detail, but the requirement is unavoidable: future attention must read the exact state corresponding to the accepted input history.

At BF16 latent-cache precision, 8,000 tokens across 61 layers require approximately