Introduction

DeepSeek-V3 co-designs model architecture, numerical precision, and distributed training around a single goal: scaling model capacity without scaling per-token compute or memory traffic at the same rate. It pairs a large pool of finely segmented Mixture-of-Experts (MoE) feed-forward networks with a compressed Multi-head Latent Attention (MLA) cache, trains with a multi-token prediction objective and fine-grained FP8 quantization, and maps the resulting workload onto a custom pipeline and all-to-all communication schedule. A multi-stage post-training pipeline—combining supervised fine-tuning, reasoning distillation, and reinforcement learning—turns the pretrained base model into a general-purpose assistant.

This book is a technical study of the DeepSeek-V3 Technical Report (arXiv:2412.19437v2, revised February 18, 2025). It covers the original V3 release described in that paper rather than subsequent point releases, the standalone DeepSeek-R1 model, or later architectural revisions, and treats benchmark numbers as historical snapshots from the paper's evaluation setup.

Each chapter derives the core equations from first principles, traces concrete numerical examples, and connects architectural equations to the hardware constraints that motivated them. You can read the chapters sequentially or jump directly to individual topics using the table below.

Contents

Reading paths

Chapters 1–3 lay the groundwork for the rest of the book. From there, readers focused on model architecture can concentrate on Chapters 4–9 and 12; training practitioners can focus on Chapters 8–20 and 23; systems engineers can read Chapters 3, 5–6, 13–18, and 21–22; and engineers evaluating V3 for deployment can focus on Chapters 1, 3, 12, 21–25, and the end-to-end request trace in Chapter 26. Core definitions are restated in context so that later chapters on FP8 quantization or expert routing remain self-contained.

Conventions and source archives

Throughout the book, a multiply-accumulate operation counts as two FLOPs. Decimal GB denotes $10^9$ bytes and GiB denotes $2^{30}$ bytes. In explicit context-length calculations, 4K = 4,096, 32K = 32,768, and 128K = 131,072 tokens, even where the paper uses those labels loosely.

The TeX source archive for 2412.19437v2 is preserved in the parent directory (main.tex for the main text, content/fp8.tex for low-precision numerics, and tables/ for benchmark results), alongside supplementary release metadata in reference/. Where the paper omits proprietary details—such as exact corpus mixtures, internal RL hyperparameters, or production kernel code—or where the released Hugging Face configuration differs slightly from the paper's prose, the text notes the discrepancy directly; Chapter 27 catalogs the source files and configuration differences in one place.

Sources: DeepSeek-V3 Technical Report, version 2. Start with Chapter 1: The Problem V3 Sets Out to Solve.

1. The Problem V3 Sets Out to Solve

Scaling a language model's parameter count, training corpus, and context length drives up costs across every part of the stack. Parameters occupy high-bandwidth GPU memory; pre-training requires quadrillions of matrix multiplications and synchronized gradient updates; long contexts bloat the key-value (KV) attention cache; and sharding the workload across thousands of accelerators introduces network bottlenecks.

DeepSeek-V3 attacks these bottlenecks simultaneously. The technical report describes a 671-billion-parameter model that activates roughly 37 billion parameters per token, trains on 14.8 trillion tokens, and completes its main pre-training, context-extension, and post-training runs in 2.788 million H800 GPU hours (Report, §1). Making sense of those numbers requires looking closely at how conditional computation, memory compression, and hardware scheduling interact.

Decoupling capacity from per-token compute

A dense Transformer applies almost all of its layer weights to every token in the sequence, tying total parameter capacity directly to per-token floating-point operations (FLOPs). A Mixture-of-Experts (MoE) architecture breaks this coupling by replacing each dense feed-forward network (FFN) with a bank of expert FFNs and routing each token to only a small subset. That allows the model to store a massive repertoire of learned transformations while keeping per-token arithmetic modest.

You can picture the expert pool as a multi-room reference library where each query visits only a handful of rooms. Total shelf space determines what the system can store, while the number of rooms visited determines the work spent per query—with the crucial caveat that MoE experts are continuous numerical transformations rather than static lookup tables, and the router re-selects experts at each of the 58 MoE layers for every single token.

Consequently, "671B total parameters with 37B active" does not mean V3 behaves like a 37B dense model from a systems perspective. The remaining 634B parameters still have to sit in GPU memory, and across a large batch of diverse tokens, nearly all 256 routed experts in a layer will be triggered by at least some tokens. A system running V3 pays the per-token compute cost of a 37B model alongside the memory footprint and network dispatch overhead of a 671B model.

Why isolated optimizations fall short

Cutting per-token FFN FLOPs via sparse routing only shifts the bottleneck elsewhere. If the network saturates while dispatching tokens to experts on remote nodes, the GPUs sit idle waiting for data. If the network keeps up, long-context KV caches can still exhaust HBM capacity during inference, or activation memory and optimizer states can force expensive tensor parallelism during training. And if activations and weights are quantized to 8-bit floating point (FP8) to speed up matrix multiplication and shrink network traffic, activation outliers can destabilize training unless quantization scales are tightly localized.

DeepSeek-V3 pairs each architectural or numerical optimization with a compensating mechanism to handle the side effects it creates:

None of these choices stands alone. Fine-grained experts are only viable because DualPipe hides the cross-node all-to-all dispatch behind computation, and because FP8 quantization halves the volume of activation traffic crossing InfiniBand and NVLink. Likewise, aggressive activation recomputation and MLA compression save enough HBM that the model can train without tensor parallelism, eliminating all-reduce synchronization inside every layer. During pre-training, batch-level bias updates keep experts balanced without distorting gradients; during inference, when real-world traffic patterns shift, the serving system deploys redundant copies of hot experts instead.

Training vs. serving workloads

Pre-training and live serving pull the hardware in opposite directions. Pre-training processes massive, uniform batches of 4,096-token sequences with a single goal: maximizing sustained token throughput per dollar while keeping optimization numerically stable. Every token in a training batch must pass through both forward and backward passes on schedule.

Live inference faces unpredictable request arrivals, variable prompt lengths, and strict latency targets—both time-to-first-token (TTFT) and inter-token latency (ITL). Waiting to accumulate a massive batch improves GPU matrix utilization during MoE execution, but directly degrades user-facing latency.

To handle both regimes, V3 disaggregates serving into separate prefill and decode stages running on differently configured GPU pools. Prefill ingests the user's entire prompt in parallel, generating enough token volume to keep expert matrix multiplications compute-bound even on smaller node groups. Decode generates one token at a time per active request; because each step reads vast expert weights for very few tokens, it is memory-bandwidth-bound and latency-sensitive, requiring a much wider pool of GPUs with high-bandwidth all-to-all interconnects.

Inherited foundations vs. new contributions

DeepSeek-V3 does not start from a blank slate; it directly inherits its two signature architectural blocks—Multi-head Latent Attention (MLA) and DeepSeekMoE—from DeepSeek-V2. What V3 adds on top of the V2 backbone are four main contributions:

1. Auxiliary-loss-free load balancing, which replaces gradient-distorting token-drop or auxiliary penalties with a dynamically adjusted per-expert routing bias.
2. Multi-Token Prediction (MTP), which attaches a sequential prediction module during training to densify supervision signals and optionally draft tokens for speculative decoding.
3. Fine-grained FP8 mixed-precision training, validated for the first time on a frontier-scale MoE model using $1 \times 128$ activation tiles and $128 \times 128$ weight blocks.
4. DualPipe and topology-aware communication kernels, which overlap bidirectional pipeline stages with cross-node InfiniBand and intra-node NVLink expert dispatch.

When reading the benchmark results later in the book, it helps to keep this lineage in mind. The paper provides controlled smaller-scale ablations isolating the gains from auxiliary-loss-free balancing, MTP, and FP8 quantization, whereas the end-to-end capability of the 671B checkpoint reflects the combined effect of the V2 architecture, the 14.8T-token corpus, the new V3 training mechanisms, and reasoning distillation from DeepSeek-R1.

Unpacking the $5.576M training cost figure

The widely cited $5.576M training cost comes from multiplying 2.788 million H800 GPU hours by a benchmark rental rate of $2.00 per GPU hour. In the paper's accounting, that total covers three specific phases of the final production run:

• Pre-training (14.8T tokens): 2,664K H800 GPU hours ($5.328M)
• Context extension (to 32K and 128K): 119K H800 GPU hours ($0.238M)
• Post-training (SFT and RL): 5K H800 GPU hours ($0.010M)

A GPU hour measures device-hours of compute: running 2,048 H800 GPUs continuously for one wall-clock hour consumes 2,048 GPU hours. At full cluster utilization, 2.664 million GPU hours translates to roughly 54.2 days of continuous wall-clock run time on a 2,048-GPU cluster.

As the authors explicitly note, this figure measures the marginal compute bill of the final successful training run. It does not include capital expenditure for the cluster, data acquisition and filtering pipelines, salary overhead, prior architectural ablations, or the compute spent training the teacher models used during post-training distillation. Chapter 25 examines this cost breakdown in detail.

Base model vs. post-trained assistant

The original V3 release consists of two distinct artifacts evaluated under different protocols: DeepSeek-V3-Base, which models raw token continuation probabilities after pre-training and long-context extension, and DeepSeek-V3 (the chat checkpoint), which undergoes supervised fine-tuning, reasoning distillation, and reinforcement learning.

When the chat model solves a competition math problem or writes working code, its output reflects the interplay of three stages: domain knowledge absorbed during the 14.8T-token pre-training run, chain-of-thought formatting and verification habits distilled during post-training, and the decoding parameters used at runtime. Chapters 23 and 24 separate the empirical evidence for the base model from the post-trained chat evaluations so we can see precisely where each stage moves the needle.

Sources: main.tex, lines 151–267 (Report, §1) for parameter counts, token budgets, and training cost tables; §3.4 for prefill/decode disaggregation; §6 for deployment characteristics. Continue to Chapter 2: The Foundations the Paper Assumes or Chapter 3: A Complete Map of the Model.

2. The Foundations the Paper Assumes

DeepSeek-V3 is an autoregressive, decoder-only Transformer whose dense feed-forward blocks are largely replaced by Mixture-of-Experts layers and whose attention mechanism compresses key-value states into low-rank latents. Before diving into V3's specific equations and systems engineering, this chapter reviews the mathematical and hardware building blocks that the technical report takes as given.

From text to token embeddings

A language model does not ingest raw strings directly; a tokenizer first segments text into a sequence of integer IDs $x_1, x_2, \ldots, x_T$, where each $x_t$ belongs to a fixed vocabulary of size $V$. Depending on how common a character sequence is, a single token might represent an entire word, a subword root, a punctuation mark, or a fallback byte. Because different models use different vocabularies and segmentation rules, token counts are only directly comparable within the same tokenizer.

At the entrance to the network, an embedding matrix maps each discrete token ID $x_t$ to a continuous vector $h_t \in \mathbb{R}^d$. The hidden dimension $d$ sets the width of the residual stream that runs through the entire model; in DeepSeek-V3, $d = 7168$. As the vector passes through successive layers, attention and feed-forward blocks add context-dependent updates directly onto this 7,168-dimensional stream.

Autoregressive factorization

An autoregressive language model factors the joint probability of a token sequence into a product of conditional next-token probabilities via the chain rule:

This single equation governs both training and inference, even though their execution patterns look completely different on hardware. During generation, the model only knows the tokens produced so far: it computes $p(x_t \mid x_1, \ldots, x_{t-1})$, samples or selects $x_t$, appends it to the input sequence, and repeats the loop one token at a time. During training, the entire ground-truth sequence $x_1, \ldots, x_T$ is already known up front, so the GPU can evaluate the conditional distributions for all $T$ positions in a single parallel forward pass—using a causal mask inside the attention layers to prevent position $t$ from peeking at positions $t+1$ and beyond.

Logits, softmax, and cross-entropy loss

After the final Transformer layer, an output projection matrix maps the normalized hidden state at position $t$ to a vector of $V$ unnormalized scores called logits, $z \in \mathbb{R}^V$. The softmax function converts these logits into a normalized probability distribution over the vocabulary:

Because shifting every logit by a constant $c$ multiplies both numerator and denominator by $\exp(c)$, software kernels subtract $\max_k z_k$ before exponentiating to avoid floating-point overflow.

During training, if the actual next token in the corpus is $y$, the model incurs a cross-entropy loss of $-\log p_y$. Assigning probability $0.5$ to the true token yields a loss of $-\ln(0.5) \approx 0.693\text{ nats}$, while assigning $0.01$ yields $4.605\text{ nats}$. Minimizing average cross-entropy over trillions of tokens teaches the model the statistical structure, facts, and reasoning patterns embedded in the training corpus. Because raw web text also contains noise, unverified claims, and unfinished fragments, a low pre-training cross-entropy does not by itself make the model a reliable conversational assistant; shaping how the model responds requires subsequent instruction tuning and reinforcement learning.

The pre-norm decoder block

Each layer in a modern decoder-only Transformer follows a two-stage pre-normalization residual layout:

The first sub-block normalizes the residual stream, mixes information across token positions up to $t$ via causal self-attention, and adds the result back to $h_t$. The second sub-block normalizes the updated state $h'_t$, applies a position-wise feed-forward network (FFN) to each token independently, and adds that output back to the residual stream.

DeepSeek-V3 preserves this exact macro-structure across all 61 layers, substituting Multi-head Latent Attention (MLA) for standard multi-head attention in every layer and replacing the dense FFN with a DeepSeekMoE block in layers 4 through 61.

RMSNorm and SwiGLU feed-forward networks

Rather than centering activations by subtracting their mean as in classic LayerNorm, DeepSeek-V3 uses Root Mean Square Normalization (RMSNorm), which scales each vector by its root-mean-square magnitude and applies a learned per-channel gain $g \in \mathbb{R}^d$:

Here $\epsilon$ is a small constant (such as $10^{-6}$) that guards against division by zero. Because divisions and square roots are sensitive to rounding error, V3 keeps RMSNorm computations in higher precision even when the surrounding matrix multiplications run in FP8.

Both the dense FFNs and individual MoE experts in V3 use the SwiGLU architecture, which gates one linear projection with the SiLU activation of another before projecting back to the residual width $d$:

where $\odot$ denotes elementwise multiplication. If the input vector $h$ has dimension $d$ and the intermediate projections $W_g h$ and $W_u h$ have dimension $f$, then each SwiGLU unit comprises three matrices—gate $W_g \in \mathbb{R}^{f \times d}$, up $W_u \in \mathbb{R}^{f \times d}$, and down $W_d \in \mathbb{R}^{d \times f}$—totaling $3df$ parameters.

Multi-head attention and the KV cache

Inside an attention layer, each token state produces query ($q$), key ($k$), and value ($v$) vectors across $n_h$ parallel heads. At step $t$, the query vector $q_t$ scores its compatibility with the key vectors $k_1, \ldots, k_t$ of all preceding tokens, turns those scores into attention weights via softmax, and computes a weighted sum of the corresponding value vectors $v_1, \ldots, v_t$.

When generating tokens one by one at inference time, the keys and values for earlier positions $1, \ldots, t-1$ never change. Rather than recomputing them on every step, serving engines store them in GPU memory in a KV cache. While model weights are static and shared across all concurrent users, the KV cache is dynamic state tied to a specific request: its memory footprint grows linearly with batch size, sequence length, layer count, and the number of bytes cached per token. Shrinking that per-token cache footprint is the primary motivation behind MLA.

Backpropagation, GEMMs, and training memory

Training a linear layer $Y = XW$ requires three general matrix multiplications (GEMMs) per step: one in the forward pass to compute activations $Y = XW$, one in the backward pass to propagate gradients to earlier layers ($\nabla_X = \nabla_Y W^\top$), and one to compute the weight gradients ($\nabla_W = X^\top \nabla_Y$). Consequently, an effective FP8 training scheme cannot just quantize forward inference; it has to handle all three GEMM directions while keeping gradient accumulation accurate.

Furthermore, because computing $\nabla_W = X^\top \nabla_Y$ during the backward pass requires the forward activations $X$, a training cluster must keep forward activations alive in memory across pipeline stages—or explicitly recompute them on the fly. On top of the model weights, gradients, and saved activations, the AdamW optimizer tracks two running statistics (first and second moments) for every parameter. A model's inference checkpoint size is therefore only a fraction of its peak training memory footprint.

Compute, memory bandwidth, and the roofline model

Whether an operation runs fast or slow on a GPU depends on the balance between three hardware limits: peak tensor-core arithmetic throughput (FLOPs/s), high-bandwidth memory (HBM) read/write speed (bytes/s), and interconnect bandwidth across NVLink or InfiniBand.

The ratio of floating-point operations performed to bytes read from memory is the operation's arithmetic intensity. A simplified roofline bound captures how arithmetic intensity dictates hardware utilization:

When processing a long prompt during prefill or a full micro-batch during training, thousands of tokens multiply the same weight matrix simultaneously. The weight bytes are read from HBM once and reused across thousands of vectors, yielding high arithmetic intensity and pushing the GPU toward its compute peak. During autoregressive decode, by contrast, only a handful of tokens may route to a given MoE expert on any single step. The GPU must stream tens of megabytes of expert weights out of HBM just to multiply a few vectors, making decode heavily memory-bandwidth-bound and communication-latency-bound. This fundamental shift in arithmetic intensity is why DeepSeek-V3 uses different cluster topologies and parallelism strategies for training, prefill, and decode.

Sources: Report §2 and §4.2 (main.tex, lines 270–455 and 714–744) for V3's Transformer structure and hyperparameters; content/fp8.tex for forward/backward GEMM and precision assignments. Continue to Chapter 3: A Complete Map of the Model.

3. A Complete Map of the Model

DeepSeek-V3's main backbone consists of 61 Transformer layers operating on a 7,168-dimensional residual stream. Every layer uses Multi-head Latent Attention (MLA) for context mixing. The first three layers use standard dense SwiGLU feed-forward networks, while the remaining 58 layers replace the dense FFN with a DeepSeekMoE block containing 1 shared expert and 256 routed experts (of which 8 are activated per token). Attached to the top of this backbone is a 1-step Multi-Token Prediction (MTP) module that provides auxiliary supervision during training and can optionally draft tokens during speculative decoding.

Rather than treating "671B total parameters and 37B active parameters" as opaque summary statistics, we can reconstruct both figures directly from the shapes of the weight matrices inside each block.

DeepSeek-V3 architecture diagram from Report §2. The hatched vectors ($c_t^{KV}$ and $k_t^R$) indicate the compressed representations stored in the KV cache during inference. While the diagram illustrates the general MoE block, the first three layers of V3 use dense FFNs instead.

Architectural hyperparameters

Table 1 consolidates the architectural dimensions reported in §4.2 alongside supplementary dimensions from the released Hugging Face configuration (reference/release-config.json):

Tracing a token through the 61-layer stack

When an input token enters the model, the embedding table maps its integer ID to a vector $h_t^{(0)} \in \mathbb{R}^{7168}$. Each layer $\ell \in \{1, \ldots, 61\}$ applies RMSNorm, projects the normalized state into compressed query and key-value latents inside MLA, mixes context from positions $\le t$, and adds the projected attention output back to the residual stream.

In layers 1 through 3, the post-attention state passes through a single wide SwiGLU FFN with intermediate dimension 18,432. In layers 4 through 61, it enters the MoE block instead: a linear router scores all 256 routed experts, selects the top 8, and evaluates those 8 experts alongside the 1 shared expert. The weighted sum of the 8 routed expert outputs is added to the shared expert output and merged back into the residual stream. Notice that the choice of 8 routed experts is completely independent across tokens and across layers—token $t$ may visit one set of 8 experts in layer 4 and an entirely different set in layer 5.

After layer 61, a final RMSNorm and the unembedding matrix project the 7,168-dimensional state onto 129,280 vocabulary logits. During standard inference, the MTP module is not needed to produce this next-token distribution; its regularization effect is already baked into the backbone weights during training.

Parameter accounting: MoE and dense FFNs

Every expert in DeepSeek-V3 is a SwiGLU network with input/output width $d = 7168$ and intermediate width $f = 2048$. Because a SwiGLU block has three matrices ($W_g, W_u \in \mathbb{R}^{f \times d}$ and $W_d \in \mathbb{R}^{d \times f}$) and no linear bias terms, a single expert contains:

Each MoE layer houses 256 routed experts plus 1 shared expert, for 257 experts total:

Multiplied across all 58 MoE layers, the expert pool accounts for:

Over 97.8% of DeepSeek-V3's total parameters live inside the MoE expert matrices. For any single token, however, only 9 experts (1 shared + 8 routed) run per MoE layer. Across all 58 MoE layers, the active expert parameters per token come to:

What about the first three dense layers? With $d = 7168$ and intermediate width $18{,}432$ (equivalent to 9 experts worth of width), each dense FFN contains:

contributing $3 \times 396.36\text{M} \approx \mathbf{1.19\text{B}}$ parameters that are both stored and active for every token.

Parameter accounting: Multi-head Latent Attention

Next, we count the linear projections inside a single MLA layer (derived in detail in Chapter 5):

Note how the output projection works: 128 heads each produce a 128-dimensional value vector, concatenating to an intermediate width of $128 \times 128 = 16{,}384$, which $W^O$ projects back down to the 7,168-dimensional residual stream. Across all 61 layers, the attention projections contain $61 \times 187.10\text{M} \approx \mathbf{11.41\text{B}}$ parameters, all of which are active for every token.

Summing the backbone and the MTP module

Finally, the vocabulary embedding matrix and final output projection head each have shape $129{,}280 \times 7{,}168 \approx 0.927\text{B}$ parameters. Because the release configuration specifies tie_word_embeddings: false, the two tables are stored separately, contributing $\mathbf{1.85\text{B}}$ parameters.

Summing the four major components of the 61-layer main model gives:

Replacing the 257 stored experts per MoE layer with the 9 active experts per token gives the active parameter count:

(Small router matrices of size $256 \times 7168 \approx 1.84\text{M}$ per MoE layer and 1D RMSNorm vectors account for the remaining ~0.1B fraction. Also note that while standard parameter-counting conventions include the embedding table in the active total, looking up a single token ID only reads one 7,168-element row rather than multiplying the full 0.927B embedding matrix.)

Why does the official Hugging Face repository list the downloadable checkpoint at 685B parameters instead of 671B? The extra ~14B parameters belong to the single-layer MTP module shipped alongside the main weights. An MTP depth of 1 adds one complete MoE Transformer layer (~11.32B in experts + ~0.19B in MLA) plus a $7168 \times 14336$ linear projection matrix (~0.10B). In the repository's weight documentation, the MTP module is listed at ~11.5B unique parameters (or ~14B when counting its references to the shared input embedding and output head), bringing the combined checkpoint to ~685B.

Paper vs. release configuration nuances

When cross-referencing the paper against the released config.json and weight documentation, three minor implementation differences stand out:

1. Maximum position embeddings: The paper describes context extension up to 128K tokens (131,072), whereas release-config.json sets max_position_embeddings: 163840 (160K). The larger buffer size in the config file sets an upper bound for RoPE table allocation, while 128K remains the benchmarked context ceiling in the report.
2. Initialization standard deviation: Section 4.2 reports that weights were initialized from a normal distribution with standard deviation $0.006$, whereas release-config.json leaves the default Hugging Face field initializer_range: 0.02. Because a pretrained checkpoint loads trained weights from disk, the config field is merely a leftover default in the modeling script.
3. Routed scaling factor: In release-config.json, the normalized router weights are multiplied by routed_scaling_factor: 2.5 before summing the 8 routed expert outputs, a constant inherited from the DeepSeekMoE implementation that is omitted from the simplified schematic equations in §2.1.2.

Sources: Report §4.2 (main.tex, lines 714–727) for layer and dimension counts; §2.1 (main.tex, lines 294–356) for MLA and DeepSeekMoE equations; reference/release-config.json and the official weight documentation for vocabulary size, dense FFN width, and MTP parameter breakdowns. Continue to Chapter 4: Attention and Rotary Position.

4. Attention and Rotary Position

Feed-forward layers transform each token's hidden state in isolation; causal self-attention is the mechanism that lets a token inspect earlier positions in the sequence, connecting a pronoun to its antecedent or a function call to an earlier import statement.

To see why DeepSeek-V3 structures Multi-head Latent Attention (MLA) the way it does, we first need to examine standard scaled dot-product attention and the linear-algebraic friction between Rotary Position Embeddings (RoPE) and low-rank key compression.

Scaled dot-product attention

Given an input state $h_t \in \mathbb{R}^d$ at token position $t$, a standard attention head projects $h_t$ into a query, a key, and a value vector:

To aggregate context at step $t$, the head takes the dot product between the current query $q_t$ and the key $k_j$ of every preceding position $j \le t$, scales by $1/\sqrt{d_k}$ (where $d_k$ is the query/key width), normalizes the scores via softmax, and takes the weighted sum of the value vectors:

For example, if the scaled dot products for three positions are $0$, $0$, and $\log 2$, their exponentials are $1, 1, 2$, yielding attention weights $1/4, 1/4, 1/2$: the third position contributes half the output vector while the first two contribute a quarter each.

Why divide by $\sqrt{d_k}$? If the entries of $q_t$ and $k_j$ have roughly zero mean and unit variance, the variance of their $d_k$-element dot product grows proportionally to $d_k$. Dividing by $\sqrt{d_k}$ keeps the variance of the pre-softmax logits near $1$ regardless of head width, preventing wide heads from saturating the softmax into near-one-hot distributions early in training. In DeepSeek-V3, each head concatenates a 128-dimensional content vector and a 64-dimensional positional vector, so the scaling denominator is $\sqrt{128 + 64} = \sqrt{192}$ (further adjusted by a YaRN temperature scale during long-context extension).

Causal masking vs. document masking

During autoregressive training, position $t$ predicts token $t+1$ and must never read information from positions $j > t$. Hardware kernels enforce this causality during parallel training by adding $-\infty$ to all upper-triangle entries ($j > t$) of the score matrix before applying softmax, zeroing out their attention weights $a_{tj}$.

When multiple shorter documents are concatenated into a single 4,096-token training sequence to avoid padding waste, an implementation can also apply a block-diagonal document mask so tokens in one document cannot attend to tokens from a previous document in the same packed sequence. Notably, DeepSeek-V3 does not apply cross-sample attention masking during pre-training packing (allowing attention to bleed across packed document boundaries), whereas it does isolate packed samples during supervised fine-tuning (Chapter 10).

Encoding relative order with RoPE

Dot-product attention over raw content vectors $q_t$ and $k_j$ is permutation-equivariant: swapping the positions of two earlier tokens leaves their dot products with $q_t$ unchanged. To make attention sensitive to token order, DeepSeek-V3 uses Rotary Position Embedding (RoPE).

RoPE partitions a vector into 2D coordinate pairs and rotates each pair in the plane by an angle proportional to the token's absolute position index $t$. For a single 2D pair with base frequency $\omega$, the rotation matrix at position $t$ is:

Applying $R(t)$ to the query and $R(j)$ to the key yields a dot product that depends only on the relative distance $j - t$:

because the transpose of a 2D rotation by $t\omega$ is a rotation by $-t\omega$, and composing rotations adds their angles. Across a $d_h^R$-dimensional vector, RoPE assigns a geometric progression of frequencies $\omega_m$ to the $d_h^R / 2$ coordinate pairs, allowing attention heads to track both fine-grained local adjacency (high frequencies) and broad document-level distance (low frequencies).

Why standard RoPE collides with low-rank KV compression

Now consider what happens when we try to compress keys into a low-rank latent vector $c_j \in \mathbb{R}^{d_c}$ so we don't have to cache full keys for all 128 heads. Suppose each head's key is reconstructed from the shared latent via a linear up-projection $k_j = U_K c_j$.

If there is no positional rotation, the dot product between query $q_t$ and key $k_j$ factors cleanly:

Because $U_K^T$ does not depend on the historical position $j$, we can multiply $U_K^T q_t$ once at step $t$ to form an absorbed query $\widetilde{q}_t = U_K^T q_t \in \mathbb{R}^{d_c}$, and then take dot products directly against the cached latents $c_j$ for all $j \le t$. We never have to materialize the expanded keys $k_j$ in memory.

If we apply standard RoPE to the reconstructed key $k_j = R(j) U_K c_j$, however, the position-dependent rotation matrix $R(j)$ sits sandwiched between the query and the up-projection:

Because $R(j)$ varies with every historical token index $j$ and does not commute with the rectangular projection $U_K$, we can no longer absorb $U_K$ into a single position-independent query vector $\widetilde{q}_t$. To evaluate attention scores, the GPU would either have to recompute $R(j) U_K c_j$ for every cached token $j$ at every decode step—destroying inference compute efficiency—or store the expanded, rotated keys $R(j) U_K c_j$ for all 128 heads, destroying the KV cache savings.

V3's solution: decoupled RoPE

DeepSeek-V2 and V3 resolve this algebraic impasse by splitting each query and key into two separate slices: a head-specific content slice ($q_{t,i}^C, k_{j,i}^C \in \mathbb{R}^{128}$) that undergoes low-rank compression without RoPE, and a narrower positional slice ($q_{t,i}^R, k_j^R \in \mathbb{R}^{64}$) that carries RoPE.

For head $i$, the complete query and key vectors are formed by concatenation:

Crucially, the 64-dimensional rotated positional key $k_j^R$ is shared across all 128 heads (acting like a multi-query positional component), while each head $i$ retains its own rotated positional query $q_{t,i}^R$. When taking the dot product, the score splits into the sum of a content term and a positional term:

Now both terms are cheap to store and evaluate:

1. The unrotated content term $(q_{t,i}^C)^T k_{j,i}^C$ has no $R(j)$ matrix in the middle, so the key up-projection $U_K$ absorbs directly into the query vector.
2. The rotated positional key $k_j^R$ cannot be absorbed, but because it is only 64 elements wide and shared across all 128 heads, caching it outright adds just 64 scalars per token per layer alongside the 512-dimensional content latent.

Values $v_{j,i}^C \in \mathbb{R}^{128}$ only carry content—position affects which values are selected via the softmax weights $a_{tj}$, so no positional value vector is needed. The next chapter steps through the full matrix algebra of this MLA pipeline.

Sources: Report §2.1.1 (main.tex, lines 294–332) for decoupled RoPE equations and attention scaling; §4.3 and Chapter 12 for YaRN long-context adjustments. Continue to Chapter 5: Deriving Multi-head Latent Attention.

5. Deriving Multi-head Latent Attention

Multi-head Latent Attention (MLA), introduced in DeepSeek-V2 and retained in DeepSeek-V3, replaces the per-head key and value projections of standard multi-head attention with a shared low-rank latent bottleneck. Rather than quantizing an existing KV cache into fewer bits per number, MLA changes the linear algebra of the attention projections so that the model caches a single 512-dimensional content latent and a 64-dimensional positional key per token, reconstructing or absorbing the 128 head-specific projections on the fly.

Dimensions of the MLA block

At layer $\ell$, let $h_t \in \mathbb{R}^{7168}$ be the normalized hidden state for token $t$. DeepSeek-V3 configures MLA with $n_h = 128$ heads and the following per-head and latent widths:

• Per-head content query/key width: $d_h^C = 128$ (concatenated across heads: $128 \times 128 = 16{,}384$)
• Per-head value width: $d_h^V = 128$ (concatenated across heads: $16{,}384$)
• Per-head RoPE query/key width: $d_h^R = 64$ (concatenated queries across heads: $128 \times 64 = 8{,}192$; shared key: $64$)
• KV content compression width: $d_c = 512$
• Query compression width: $d'_c = 1{,}536$

In the actual architecture (§4.2), V3 applies an extra RMSNorm layer to the compressed latents $c_t^Q$ and $c_t^{KV}$ immediately after their down-projections. In the derivations below, $c_t^Q$ and $c_t^{KV}$ denote the latent vectors after this RMSNorm has been applied; because the KV cache stores the post-RMSNorm latent $c_t^{KV}$, all downstream up-projections remain purely linear maps that can be algebraically absorbed during inference.

Joint low-rank KV compression and decoupled key RoPE

To produce keys and values, MLA first projects $h_t$ down into a 512-dimensional latent vector (followed by RMSNorm):

From this single vector $c_t^{KV}$, each head $i \in \{1, \ldots, 128\}$ generates its 128-dimensional content key and 128-dimensional value via head-specific linear up-projections:

Stacking $U^K_1, \ldots, U^K_{128}$ forms the full content-key up-projection matrix $W^{UK} \in \mathbb{R}^{16384 \times 512}$, and stacking $U^V_1, \ldots, U^V_{128}$ forms $W^{UV} \in \mathbb{R}^{16384 \times 512}$.

In parallel, a separate projection maps $h_t$ directly to the shared 64-dimensional positional key and applies RoPE for position $t$:

During autoregressive decoding, only $c_t^{KV} \in \mathbb{R}^{512}$ and $k_t^R \in \mathbb{R}^{64}$ are written to the KV cache—a total of $512 + 64 = 576$ scalars per token per layer.

Low-rank query compression

MLA also compresses the queries through a 1,536-dimensional bottleneck (again followed by RMSNorm):

From $c_t^Q$, each head reconstructs both its 128-dimensional content query and its 64-dimensional rotated positional query:

where $U^Q_i \in \mathbb{R}^{128 \times 1536}$ (stacking to $W^{UQ} \in \mathbb{R}^{16384 \times 1536}$) and $U^R_i \in \mathbb{R}^{64 \times 1536}$ (stacking to $W^{QR} \in \mathbb{R}^{8192 \times 1536}$).

Why compress queries if queries are never stored in the KV cache? First, replacing a direct $7168 \to 24576$ projection ($176.16\text{M}$ parameters) with $7168 \to 1536 \to 24576$ ($11.01\text{M} + 37.75\text{M} = 48.76\text{M}$ parameters) trims roughly $127\text{M}$ parameters per layer. Second, during training, caching the 1,536-dimensional query latent requires far less activation memory than keeping the 24,576-dimensional expanded queries alive for the backward pass.

The expanded attention equations

If we materialize all head vectors explicitly—as is typically done during training and large-batch prefill—the attention score between query position $t$ and key position $j \le t$ for head $i$ is:

Applying softmax over $j \le t$ gives the attention weights $a_{tji}$, which combine the head values:

During decode, however, materializing $k_{j,i}^C$ and $v_{j,i}^C$ for all past tokens $j \le t$ and all 128 heads would force us either to store 32,768 numbers per token in the KV cache or to re-run the $U^K_i$ and $U^V_i$ matrix multiplications over the entire history at every single token step. Matrix absorption avoids both costs.

Matrix absorption for inference

1. Absorbing the key up-projection $U^K_i$

Substitute $k_{j,i}^C = U^K_i c_j^{KV}$ into the content score term in $z_{tji}$:

For the current token $t$, we compute the absorbed query for head $i$:

Now, for every historical position $j \le t$, the content dot product for head $i$ is simply $\widetilde{q}_{t,i}^T c_j^{KV}$—an inner product between head $i$'s 512-dimensional absorbed query and the single shared 512-dimensional latent $c_j^{KV}$ stored in the cache. All 128 heads read the exact same cached vector $c_j^{KV}$ from HBM.

2. Absorbing the value up-projection $U^V_i$

Next, substitute $v_{j,i}^C = U^V_i c_j^{KV}$ into the value aggregation sum:

Because $U^V_i$ is constant with respect to the historical index $j$, it factors outside the sum. Each head $i$ first computes a weighted sum of the 512-dimensional cached latents:

and then multiplies the single resulting vector $\widetilde{o}_{t,i}$ by $U^V_i \in \mathbb{R}^{128 \times 512}$ (or directly by the fused output matrix $W^O_{:, i} U^V_i \in \mathbb{R}^{7168 \times 512}$) once at the end of the attention step.

To see the identity on concrete numbers, take a 2D latent with $c_1^{KV} = [1, 0]^T$, $c_2^{KV} = [0, 3]^T$, attention weights $a = [0.25, 0.75]$, and a $1 \times 2$ value projection $U^V = [2, -1]$:

• Unabsorbed path: $v_1 = 2(1) - 1(0) = 2$ and $v_2 = 2(0) - 1(3) = -3$, so $o = 0.25(2) + 0.75(-3) = -1.75$.
• Absorbed path: $\widetilde{o} = 0.25[1, 0]^T + 0.75[0, 3]^T = [0.25, 2.25]^T$, and $o = U^V \widetilde{o} = 2(0.25) - 1(2.25) = -1.75$.

In exact arithmetic the two paths produce identical results; in finite precision, they differ only by floating-point rounding order.

How MLA differs from GQA and MQA

In Multi-Query Attention (MQA) or Grouped-Query Attention (GQA), heads within a group share identical key and value vectors, which restricts every head in the group to reading the exact same value subspace. In MLA, all 128 heads share the underlying 512-dimensional cached vector $c_j^{KV}$, but each head applies its own learned rank-128 readout matrices $U^K_i$ and $U^V_i$. Thus, MLA achieves a smaller cache footprint than 8-group GQA while preserving 128 distinct per-head key and value projections.

Keep in mind what MLA does and does not compress: it shrinks the per-token channel width of the KV cache from 32,768 scalars down to 576 scalars, but the cache still stores one 576-element entry for every historical token $j \le t$. Cache memory still scales linearly with sequence length $S$, and full causal attention still evaluates all $j \le t$ pairs. The next chapter works out the exact memory and FLOP economics of this tradeoff.

Sources: Report §2.1.1 (main.tex, lines 294–332) for the MLA equations and §4.2 (main.tex, lines 716–726) for layer dimensions and latent RMSNorm. See also the MLA class (naive vs. absorb modes) in the official reference inference code. Continue to Chapter 6: The Economics of Attention Memory.

6. The Economics of Attention Memory

During long-context generation, the request-specific KV cache can easily rival or exceed the memory footprint of the model weights on a serving node. By compressing 128 heads of keys and values into a 512-dimensional content latent plus a 64-dimensional positional key, MLA drastically cuts the bytes read from HBM on every decode step—while simultaneously changing the arithmetic intensity and FLOP profile of the attention kernel.

Exact KV-cache byte accounting

If a model with $L$ layers caches $c$ scalar elements per token per layer at $b$ bytes per element, the raw KV cache size for a single sequence of length $S$ is:

In DeepSeek-V3, $L = 61$ main layers and $c = d_c + d_h^R = 512 + 64 = 576$ elements per layer. Stored in BF16 ($b = 2$ bytes), each retained token consumes:

Across the standard context milestones reported in the paper, the raw BF16 cache footprint for a single sequence is:

At 8.58 GiB for a full 128K-token context (before paged-attention metadata, alignment padding, or activation buffers), a serving cluster can hold dozens of long-context sessions in HBM alongside the sharded MoE weights. Quantizing the cached latents to FP8 ($b = 1$ byte plus block scaling metadata) halves this footprint again to roughly 4.3 GiB at 128K.

Comparing MLA against MHA, GQA, and MQA

How does 576 scalars per token per layer compare against standard attention architectures with the same 128 heads and $d_h = 128$ content/value width?

Relative to full 128-head MHA, MLA shrinks the KV cache by nearly $57\times$ (and if the MHA baseline also cached a 64-dimensional RoPE key slice per head, its per-layer width would be $128 \times (192 + 128) = 40{,}960$, a $71.1\times$ reduction). Relative to an 8-head GQA layout common in large dense models, MLA uses $3.56\times$ less KV memory while allowing every head to maintain its own rank-128 key and value subspace rather than forcing 16 query heads to share a single static KV head.

The FLOP vs. memory-bandwidth tradeoff in absorbed MLA

While matrix absorption slashes KV cache memory traffic, it actually increases the number of floating-point operations per historical token inside the core attention loop.

To see why, count the multiply-accumulate FLOPs (2 FLOPs per multiply-add) per query head per historical token $j$:

• Unabsorbed (explicit head KV): Each head computes a dot product of width $128 + 64 = 192$ for the score ($2 \times 192 = 384$ FLOPs) and accumulates a 128-dimensional value vector ($2 \times 128 = 256$ FLOPs), totaling: $$2(128+64) + 2(128) = \mathbf{640\text{ FLOPs/head/token}}.$$
• Absorbed (latent KV): Each head computes a dot product between its 512-dimensional absorbed query $\widetilde{q}_{t,i}$ and the 512-dimensional latent $c_j^{KV}$, plus the 64-dimensional RoPE dot product ($2 \times (512 + 64) = 1{,}152$ FLOPs), and accumulates the 512-dimensional latent vector into $\widetilde{o}_{t,i}$ ($2 \times 512 = 1{,}024$ FLOPs), totaling: $$2(512+64) + 2(512) = \mathbf{2{,}176\text{ FLOPs/head/token}}.$$

Why is a $3.4\times$ increase in inner-loop FLOPs a massive win during autoregressive decoding? Look at the arithmetic intensity across all 128 heads:

• In standard 128-head MHA, reading 32,768 BF16 scalars (65,536 bytes) per historical token to perform $128 \times 640 = 81{,}920$ FLOPs yields an arithmetic intensity of just $1.25\text{ FLOPs/byte}$—catastrophically memory-bandwidth-bound on an H800 GPU whose tensor-core-to-HBM balance point is hundreds of FLOPs per byte.
• In absorbed MLA, all 128 query heads share the same 576 cached scalars (1,152 BF16 bytes) per historical token. Once a GPU streaming multiprocessor loads $c_j^{KV}$ and $k_j^R$ from HBM into SRAM, it reuses those 1,152 bytes across all 128 heads simultaneously to perform $128 \times 2{,}176 = 278{,}528$ FLOPs, boosting arithmetic intensity to $241.8\text{ FLOPs/byte}$.

Why prefill and decode use different MLA execution paths

This difference in arithmetic intensity explains why the official DeepSeek-V3 reference implementation (inference/model.py) provides two distinct execution modes—and why production serving engines switch strategies between prefill and decode:

1. Prefill / Training (unabsorbed expansion): When processing a 4,096-token prompt in parallel, the up-projections $W^{UK} c^{KV}$ and $W^{UV} c^{KV}$ are large dense matrix multiplications, and the core attention loop has $640$ FLOPs per head per pair instead of $2{,}176$. Because the expanded KV tensors are consumed immediately across thousands of query positions in SRAM (via FlashAttention tiling), expanding $k^C$ and $v^C$ avoids the $3.4\times$ pairwise FLOP penalty during prefill while only writing the compact 576-element vectors ($c_t^{KV}, k_t^R$) out to the persistent HBM cache for subsequent decoding.
2. Decode (absorbed latent attention): When generating one token per sequence, matrix absorption (absorb mode) keeps the KV cache at 576 elements per token and converts the memory-bound KV scan into a compute-efficient batched matrix multiply over the shared latents in SRAM.

Even with MLA, causal prefill still performs $O(S^2)$ pairwise token comparisons across the prompt, and each decode step still scans $O(S)$ historical latents. MLA removes the KV memory wall that previously capped long-context batch sizes, shifting the remaining long-context bottleneck from HBM capacity to attention compute.

Sources: Report §2.1.1 (main.tex, lines 294–332) and §4.2 (main.tex, lines 716–726); naive vs. absorb modes in inference/model.py. See Chapter 21 for how prefill and decode clusters manage KV cache and expert parallelism in production. Continue to Chapter 7: Fine-grained and Shared Experts.

7. Fine-grained and Shared Experts

In DeepSeek-V3, an "expert" is not a standalone domain-specific language model or an external agent; it is a single SwiGLU feed-forward network (FFN) living inside a Transformer layer. In 58 of the model's 61 layers (layers 4 through 61), the dense FFN is replaced by a DeepSeekMoE block containing 1 shared expert and 256 routed experts. For each token, a learned linear router selects 8 of the 256 routed experts, evaluates them in parallel alongside the shared expert, and sums their outputs back into the residual stream.

Anatomy and FLOP cost of a single expert

Let $u_t \in \mathbb{R}^{7168}$ be the normalized post-attention hidden state for token $t$. Each expert $i$—whether shared or routed—is a three-matrix SwiGLU transformation with input/output dimension $d = 7168$ and intermediate hidden dimension $f = 2048$:

As we calculated in Chapter 3, each expert holds $3 \times 7168 \times 2048 \approx 44.04\text{M}$ parameters. Counting two FLOPs per multiply-accumulate across the three matrix-vector products, evaluating one expert for a single token requires:

Because each token visits 1 shared expert and 8 routed experts (9 experts total), its active FFN compute per MoE layer is:

which exactly matches the FLOP cost of a single dense SwiGLU FFN in layers 1–3 ($f_{\text{dense}} = 9 \times 2048 = 18{,}432$). By contrast, evaluating all 257 stored experts in a layer would cost $257 \times 88.08\text{M} \approx 22.64\text{B}$ FLOPs per token. Sparse routing thus provides the parameter capacity of 257 experts for the per-token compute budget of 9.

Why DeepSeekMoE uses fine-grained and shared experts

Conventional MoE designs (such as Mixtral $8\times7$B or GShard) typically use a small number of full-width experts (e.g., 8 or 16 experts, picking top-1 or top-2). DeepSeekMoE modifies that blueprint in two ways:

1. Fine-grained expert segmentation: Instead of storing 32 full-sized experts of width $f = 16{,}384$ and activating 1, DeepSeek-V3 slices the intermediate dimension into $8\times$ narrower units ($f = 2048$), creating 256 experts and activating 8 per token. Keeping the total stored and active parameter counts constant while splitting experts finer increases the number of possible expert combinations from $\binom{32}{1} = 32$ to $\binom{256}{8} \approx 4.09 \times 10^{14}$. That combinatorial flexibility allows the router to compose distinct subsets of features for nuanced inputs rather than forcing a single monolithic expert to handle everything.
2. Dedicated shared expert isolation: Every token needs certain common linguistic and structural transformations regardless of domain. In a purely routed MoE, dozens of routed experts end up learning redundant copies of these shared operations, wasting parameter capacity. Dedicating one always-active expert ($N_s = 1$) to capture common cross-token transformations frees the 256 routed experts to specialize in rarer, conditional patterns.

Sigmoid affinity scoring and top-k gating

To decide which routed experts to run for token $t$, the router computes the dot product between the token's hidden state $u_t \in \mathbb{R}^{7168}$ and a learned centroid vector $e_i \in \mathbb{R}^{7168}$ for each routed expert $i \in \{1, \ldots, 256\}$. Unlike DeepSeek-V2, which applied a softmax over all 256 expert logits, DeepSeek-V3 applies an independent sigmoid activation to each logit (§2.1.2):

Why switch from softmax to sigmoid plus top-$k$ normalization? Under a 256-way softmax, experts compete globally before selection, which can compress score differences or make top-8 selection sensitive to the tail of the 248 unselected logits. With sigmoid scoring, each expert's raw affinity $s_{i,t} \in (0, 1)$ is evaluated independently.

Once the top-8 set $S_t$ is chosen (incorporating the load-balancing biases and node constraints covered in Chapter 8), the router normalizes the raw sigmoid affinities only across the 8 selected experts:

In the paper's schematic presentation, the layer output adds the residual stream, the shared expert output, and the weighted sum of the selected routed experts:

where, in the released checkpoint (reference/release-config.json and inference/model.py), the normalized weights include a fixed scalar multiplier $g_{i,t} = \alpha \cdot g'_{i,t}$ with routed_scaling_factor $\alpha = 2.5$.

For a quick two-of-four numerical example: suppose four experts receive raw sigmoid affinities $s = [0.8, 0.6, 0.3, 0.1]$. Selecting the top 2 picks experts 1 and 2, whose normalized weights are $g'_1 = 0.8 / (0.8 + 0.6) \approx 0.571$ and $g'_2 = 0.6 / 1.4 \approx 0.429$. If their scalar outputs are $10$ and $-4$, the unscaled routed sum is $0.571(10) + 0.429(-4) = 4.0$ (or $2.5 \times 4.0 = 10.0$ with the 2.5 scaling factor), which is then added to $u_t + E_{\mathrm{shared}}(u_t)$.

How gradients train a discrete router

Because choosing the top-8 indices $S_t$ is a discrete operation, how does backpropagation update the router centroids $e_i$? Look at the weighted sum $\sum_{i \in S_t} g_{i,t} E_i(u_t)$: for the 8 selected experts $i \in S_t$, the gating weight $g_{i,t}$ multiplies $E_i(u_t)$ and is a smooth, differentiable function of the sigmoid affinities $s_{j,t}$ ($j \in S_t$) and hence of the dot products $u_t^T e_j$.

If expert $i \in S_t$ produces an output vector $E_i(u_t)$ that reduces the downstream loss, the gradient with respect to $g_{i,t}$ pushes $s_{i,t}$ upward (and the competing selected affinities slightly downward via the normalization denominator), teaching the centroid $e_i$ to align more strongly with $u_t$. Unselected experts ($i \notin S_t$) have $g_{i,t} = 0$ and receive zero gradient on that token—which is why an unbalanced router that never picks certain experts will starve them of gradient updates altogether unless a load-balancing mechanism intervenes.

Per-token sparsity vs. batch-level expert activation

A single token activates only $8 / 256 = 3.125\%$ of the routed experts in a layer. Across a batch of $B$ tokens, however, each token makes $8$ selections, resulting in $A = 8B$ total expert assignments. Even under a simplified uniform-selection model, after $A$ independent draws the expected number of distinct experts touched out of 256 is:

With just $B = 32$ tokens ($A = 256$ assignments), over 63% of the 256 experts are already active; by $B = 128$ tokens ($A = 1{,}024$ assignments), more than 98% of all experts in the layer are triggered by at least one token.

This creates a fundamental tension for hardware execution:

• During training and prefill, a batch contains thousands of tokens. Every expert receives dozens or hundreds of tokens, turning each expert evaluation into a compute-efficient dense matrix-matrix multiplication (grouped GEMM).
• During small-batch autoregressive decode, if 64 active sequences generate 1 token each ($A = 512$ total assignments spread across 256 experts), each expert receives an average of only 2 tokens. The GPU cluster has to read almost all 657B expert parameters from HBM every single decode step just to multiply tiny 2-row matrices.

Fine-grained MoE therefore shifts the systems challenge from raw FLOPs to token regrouping (all-to-all dispatch), load balancing, and wide expert parallelism. Chapter 8 examines how DeepSeek-V3 balances expert load and constrains network hops without hurting model quality.

Sources: Report §2.1.2 (main.tex, lines 334–356) for DeepSeekMoE and sigmoid gating equations; §4.2 (main.tex, lines 716–727) for expert counts and dimensions; reference/release-config.json and Gate in inference/model.py for routed_scaling_factor: 2.5. Continue to Chapter 8: Routing and Load Balancing.

8. Routing and Load Balancing

Left to optimize language modeling loss alone, an MoE router frequently suffers from routing collapse: it funnels the vast majority of tokens into a handful of favored experts while leaving the rest idle. That creates a double failure: the underused experts stop receiving gradients and waste parameter capacity, while the overloaded experts become stragglers that stall distributed all-to-all execution. Conversely, if a load-balancing penalty forces every sequence to distribute tokens uniformly across all 256 experts, experts cannot specialize by domain.

DeepSeek-V3 resolves this tension with an auxiliary-loss-free balancing controller that adjusts per-expert selection biases dynamically based on batch-level load, paired with a tiny sequence-level auxiliary penalty ($\alpha = 0.0001$) and a hard 4-node communication cap per token.

The flaw in conventional auxiliary balancing losses

Standard MoE architectures encourage uniform routing by adding an explicit load-balancing penalty to the training loss:

Because $\mathcal{L}_{\mathrm{Bal}}$ is differentiated with respect to the router parameters (and indirectly affects the upstream hidden representations), choosing the coefficient $\alpha$ forces an awkward compromise. If $\alpha$ is too small, routing remains skewed and hardware utilization suffers; if $\alpha$ is large enough to enforce uniform hardware loads, the balancing gradients pull the router away from the expert choices that minimize language modeling loss. Worse still, when $\mathcal{L}_{\mathrm{Bal}}$ is computed per sequence, a pure Python file or a pure math proof is penalized for routing primarily to code or math experts, actively suppressing domain specialization.

Decoupling expert selection from mixture weighting

DeepSeek-V3 removes the primary load-balancing pressure from the gradient path altogether. Instead, each routed expert $i \in \{1, \ldots, 256\}$ maintains a non-differentiable scalar bias $b_i$ that is added to the raw sigmoid affinity $s_{i,t} = \operatorname{sigmoid}(u_t^T e_i)$ only when selecting the top-8 experts:

Once the winning index set $S_t$ is determined, the gating weights that scale the expert outputs use the unbiased original affinities $s_{i,t}$:

Why keep $b_i$ out of $g'_{i,t}$? Because $b_i$ tracks hardware queue pressure rather than semantic relevance. Adding $b_i$ during top-$k$ selection nudges marginal tokens toward underloaded experts so all 256 experts stay busy, while computing $g'_{i,t}$ from the pure affinity $s_{i,t}$ ensures that an expert admitted only because of a high positive bias does not receive an artificially inflated weight in the layer output.

To see the difference concretely, consider a top-2 router over four experts with raw affinities $s_t = [0.80, 0.70, 0.60, 0.10]$. Without biases, experts 1 and 2 win. Now suppose expert 2 is overloaded ($b_2 = -0.20$) and expert 3 is underloaded ($b_3 = +0.05$), shifting the selection scores to $s_t + b = [0.80, 0.50, 0.65, 0.10]$. The router now picks experts 1 and 3, but combines their outputs using their raw affinities $0.80$ and $0.60$, giving normalized weights $0.80 / 1.40 \approx 0.571$ and $0.60 / 1.40 \approx 0.429$.

Iterative bias feedback during pre-training

At the end of each training step, the system tallies how many tokens across the global batch were routed to each expert $i$. If expert $i$'s observed load $\ell_i$ exceeds the uniform target load $\ell_{\mathrm{target}}$, its bias $b_i$ is decremented by a step size $\gamma$; if $\ell_i$ falls below $\ell_{\mathrm{target}}$, $b_i$ is incremented by $\gamma$:

This sign-based proportional feedback loop acts like a thermostat on expert occupancy. In DeepSeek-V3 (§4.2), the bias update rate is set to $\gamma = 0.001$ for the first 14.3T pre-training tokens, and then set to $\gamma = 0.0$ for the final 500B tokens so that the late-stage learning-rate decay refines weights against a frozen routing bias distribution.

The complementary sequence-wise balance loss

Although the paper calls this approach "auxiliary-loss-free," V3 does retain a very small complementary sequence-wise balance loss to guard against pathological cases where a single individual sequence routes every single one of its tokens to the exact same experts (§2.1.2):

where $N_r = 256$, the loss coefficient is $\alpha = 0.0001$, and $f_i$ and $P_i$ measure selection frequency and normalized affinity mass across the $T$ tokens of a single sequence:

Under perfectly uniform routing, $f_i = 1$ and $P_i = 1/N_r$, making $\sum_i f_i P_i = 1$. Because $\alpha = 0.0001$ is orders of magnitude smaller than standard MoE auxiliary loss weights (which typically sit around $10^{-2}$), $\mathcal{L}_{\mathrm{Bal}}$ barely perturbs the language modeling gradients while still preventing extreme single-sequence hot-spots.

Why batch-level balancing unlocks specialization

A key insight from the paper's ablations (§4.5.3) is why the auxiliary-loss-free controller improves validation perplexity and downstream benchmarks. A global training batch mixes thousands of sequences from diverse domains—code, mathematics, English prose, Chinese web text. When balance is enforced at the global batch level (as $b_i$ does), code sequences are free to route heavily to one subset of experts while prose sequences route to another, so long as the aggregate utilization across the entire batch averages out.

In fact, when the authors tested a standard differentiable auxiliary loss computed at the batch level rather than the sequence level on 1B and 3B MoE models, it achieved the exact same validation loss as the bias-based method ($2.253$ vs. $2.258$ at 1B, and $2.080$ vs. $2.085$ at 3B). Moving the balancing scope from the individual sequence to the global batch is therefore a major driver of the specialization shown in Figure 1, while the bias update mechanism provides a clean, hyperparameter-insensitive way to enforce that batch balance at scale.

Relative expert load across domains from Report §4.5.3 (measured on 16B ablation models with 64 routed experts). Compared to sequence-level auxiliary loss (left), the auxiliary-loss-free bias controller (right) allows sharp domain-specific expert concentration on Wikipedia, GitHub, and math texts while keeping global batch load balanced.

Node-limited routing ($M = 4$)

During pre-training, DeepSeek-V3 uses 64-way Expert Parallelism (EP) spread across 8 compute nodes (8 GPUs per node, 4 routed experts per GPU). If a token's 8 selected experts could land on all 8 nodes arbitrarily, every token would have to be broadcast across the inter-node InfiniBand fabric to up to 8 different machines.

To bound cross-node InfiniBand traffic, V3 restricts each token to at most $M = 4$ nodes (§2.1.2). The 256 routed experts are partitioned into 8 node groups of 32 experts each. For each node group $g \in \{1, \ldots, 8\}$, the router sums the highest $K_r / M = 8 / 4 = 2$ candidate scores within that group (using the biased scores $s_{i,t} + b_i$ in the reference Gate implementation). It selects the 4 node groups with the highest top-2 sums, masks out all experts on the remaining 4 nodes, and then selects the global top-8 experts from the $4 \times 32 = 128$ eligible candidates. (Note that a selected node is not forced to supply exactly 2 final experts; it could supply 1, 2, 3, or more, provided all 8 winning experts live on the 4 winning nodes.)

Because the bias controller keeps expert loads tightly balanced and the 4-node cap bounds network fan-out, DeepSeek-V3 never drops tokens during either training or inference. When live serving traffic diverges from the pre-training domain mix—for instance, during a spike in coding requests that hammers specific experts—the inference engine preserves exact routing semantics by replicating hot experts across GPUs rather than altering the learned bias values (Chapter 21).

Sources: Report §2.1.2 (main.tex, lines 358–397) for routing biases, $\mathcal{L}_{\mathrm{Bal}}$, and node-limited routing; §4.2 (main.tex, lines 737–744) for the $\gamma$ schedule; §4.5.3 (main.tex, lines 912–932) for batch-vs-sequence ablations; Gate class in inference/model.py. Continue to Chapter 9: Multi-token Prediction as Training.

9. Multi-token Prediction as Training

In standard autoregressive training, the hidden state at position $i$ only receives direct cross-entropy supervision from a single immediate target: token $x_{i+1}$. DeepSeek-V3 augments this objective with a sequential Multi-Token Prediction (MTP) module that asks each position $i$ to also predict $x_{i+2}$ conditioned on the main model's hidden state and the ground-truth embedding of $x_{i+1}$.

MTP serves two distinct purposes across the lifecycle of the model: during pre-training, it acts as an auxiliary training objective that backpropagates denser supervision into the 61-layer main backbone; during inference, it can optionally be retained as a lightweight draft head for speculative decoding (Chapter 22) or discarded entirely so the main model runs with zero extra overhead.

Sequential vs. independent multi-token prediction

Earlier multi-token prediction architectures (such as Gloeckle et al., 2024) attach $D$ parallel, independent heads directly to the backbone hidden state $h_i^0$, asking $h_i^0$ to simultaneously guess $x_{i+1}, x_{i+2}, \ldots, x_{i+D+1}$ without knowing which intermediate token $x_{i+1}$ was actually chosen. Because natural language is highly multimodal—after the word "The", thousands of nouns are plausible for $x_{i+1}$, and the valid verb at $x_{i+2}$ depends entirely on which noun appeared at $x_{i+1}$—forcing a single representation to predict $x_{i+2}$ blindly mixes incompatible futures.