Introduction

Multi-Teacher On-Policy Distillation separates two difficult jobs: developing strong domain specialists and integrating their capabilities into one model. The student practices on its own generated trajectories, and the relevant specialist supplies dense next-token supervision. That simple organizing idea leads to important questions about probability, distribution shift, optimization, teacher compatibility, and compute.

This book develops those questions from first principles. It is an extended technical report with 26 chapters, worked derivations, benchmark tables, numerical examples, implementation pseudocode, an evidence audit, a reference, and exercises with solutions. It aims to give each topic enough context to be read independently while building a continuous explanation across the whole book.

Two papers, two meanings of MOPD

The main subject is Wenhan Ma and coauthors' Multi-Teacher On-Policy Distillation, arXiv:2606.30406v1. The second supplied paper, by Weichen Yu and coauthors, is Multi-Rollout On-Policy Distillation, arXiv:2605.12652v2. They are distinct methods sharing one acronym. This book calls them MT-MOPD and MR-MOPD to avoid confusion. Multi-Teacher paper, Multi-Rollout paper.

MT-MOPD changes which teacher supplies the target: a task is routed to its domain specialist. MR-MOPD changes what the teacher sees: successful and failed attempts on the same prompt provide privileged context. The book centers MT-MOPD and gives MR-MOPD a full treatment, including its distinct objectives, experiments, limitations, and unresolved source details.

Intended audience and prerequisites

The book is for readers who want to understand the methods well enough to explain, assess, or implement them. Familiarity with basic machine learning helps, but the opening chapters introduce tokens, policies, supervised fine-tuning, reinforcement learning, KL divergence, and teacher forcing.

You do not need to have read either paper first. The equations are accompanied by explanations and small examples. The implementation chapters distinguish a conceptual blueprint from an executed reproduction. The evidence chapters preserve raw results and explain how headline metrics are constructed.

Contents

Reading paths

For a first complete reading, follow Chapters 1–26. Chapters 1–5 establish the background, 6–14 develop MT-MOPD, 15–19 develop MR-MOPD, and 20–26 connect the ideas through practice, evaluation, limitations, and reference material.

For the mathematics, read 2, 4–5, 7–9, 20, 23, and 26. Pay particular attention to the distinction between fixed-context gradients and complete trajectory-distribution gradients.

For training and systems, read 3, 6, 8–12, 15–17, and 21–22. These chapters explain teacher development, target services, masks, peer availability, and the cost model.

For an evidence-focused reading, start with 1, then read 13–14 and 18–19, followed by 22–24. The audit chapter is essential if you reuse the Multi-Rollout chemistry comparisons or teacher-signal metrics.

For a short conceptual route, read 1, 4, 6, 15, 20, and 24. This route explains what changes in each method and how a training step works, while leaving the detailed derivations and tables available for later study.

How claims are handled

The book distinguishes reported paper evidence, mathematical derivations, interpretations, and proposed extensions. Standard definitions are background. An authored numerical example is not a measured model output. A proposed combination of teacher routing and peer context is not an experiment validated by the supplied papers.

Several details deserve more care than an abstract provides. The policy-gradient formula has a fixed-context derivation but should not be silently treated as the total gradient through every future visited state. The corrected top-k objective fixes a selected-coordinate optimum without becoming full-vocabulary KL. The headline MT score measures specialist-over-SFT headroom rather than raw accuracy.

MR's source also contains an inconsistent chemistry baseline, incomplete definitions for averaging metrics over constant-label prompts, and operational ambiguities around missing peers and context budgets. These issues are documented with source locators. They are neither hidden nor treated as reasons to discard all results.

Source and verification boundaries

The report is based on the full versioned TeX archives retrieved on October 13, 2026, with bibliographic metadata checked against arXiv. Textual source snapshots and SHA-256 hashes are retained under the sibling _sources/ directory. Chapter 25 maps equations, tables, and settings to exact retained files.

The book does not claim independent reproduction of either training run. It has not executed or audited the linked MR code repository. Further-reading links are primary-source pointers, not claims that every referenced work received the same full review as the two supplied papers.

Navigation links use the documentation site's routes, and each chapter includes previous/next navigation. All numerical examples use natural logarithms. Decimal KB/MB/GB are used unless MiB or another binary unit is explicitly named.

Begin with Chapter 1.

1. Two Papers, One Acronym, Two Problems

A useful language model must do several things at once. It may need to solve a mathematical problem, follow an exact formatting instruction, repair a repository, and call a tool with the right arguments. Improving each ability separately is already difficult. Making one model retain all those improvements introduces a second problem: capability integration.

Multi-Teacher On-Policy Distillation offers a way to separate these jobs. First develop specialists. Then let a generalist student practice its own responses while the appropriate specialist supplies detailed supervision. The distinction between practicing and copying is central: the student generates the paths on which it is trained.

This book studies that method and a related method that shares its acronym. It starts with probability and post-training, builds the distillation mathematics, examines the experiments, and ends with implementation, evaluation, and open questions. Each chapter reintroduces the concepts needed for its subject, so it can also be read independently.

The naming collision

The supplied links point to two distinct papers:

“MT-MOPD” and “MR-MOPD” are editorial abbreviations for this book. They are not new algorithms or the papers' official method names. When a chapter says simply “multi-teacher distillation,” it refers to the first row. This naming rule avoids conflating two different experimental programs. MT paper, MR paper.

The shared foundation is on-policy distillation: sample a trajectory from the current student and ask a teacher to score next-token distributions on its prefixes. The papers modify different parts of that foundation. MT-MOPD changes which teacher supplies the target. MR-MOPD changes what information the teacher sees.

Why specialist improvements do not simply add together

Imagine three copies of a broadly trained model. One becomes better at mathematics through verified-answer reinforcement learning. Another improves instruction following through constraint checks. A third learns repository repair through execution and tests.

Their improved behavior lives in neural-network weights. It is tempting to average those weights. But a weight vector is not a list of independent skills. The same parameters participate in many behaviors. Averaging two individually successful adaptations can produce behavior that neither model intended.

It is equally tempting to train a single model on all three rewards. This can work, but it places different domains inside one optimization process. A change in rollout length, reward scale, dataset difficulty, or learning rate may benefit one domain and harm another. A software environment that takes minutes to evaluate also complicates a loop containing short instruction-following tasks.

A third option is sequential training. That avoids some simultaneous coordination, but later updates may overwrite earlier improvements. There is no general rule that training math after instruction following preserves instruction following, or that the opposite order is better.

These are motivating possibilities, not impossibility theorems. Mixed training, sequential training, and parameter merging can be effective. The research question is whether a different organization can provide a better tradeoff under a specified recipe.

The central move: integrate behavior on student states

A specialist need not generate the student's entire answer to teach it. Suppose the student is halfway through an incorrect proof. The math teacher can read that exact prefix and assign low probability to a misleading next step and higher probability to a repair.

The student then receives information at the location it actually reached. It does not merely observe a correct proof that followed a completely different path. Repeating this procedure creates a moving curriculum: as the student changes, the distribution of prefixes on which it receives supervision changes too.

For software engineering, a prefix may include a command, a tool result, and a proposed patch. The relevant teacher reads this student-produced interaction history. It supplies a local distribution over what the student might do next. The environment still determines what a command actually returns; teacher likelihood does not rewrite the environment.

MT-MOPD applies this pattern across domains. A task's domain selects its teacher. The final student remains one model; the method does not require consulting all specialist teachers at deployment.

Why peer attempts add another kind of information

Now consider several student attempts at the same coding problem. One handles a boundary condition correctly; another fails because it confuses two identifier spaces. A scalar verifier tells us which attempt passes. It does not necessarily identify the mistaken composition of identifiers.

MR-MOPD places labeled peer attempts in the teacher's context. A successful peer can demonstrate the intended relation. A failed peer can show a plausible misunderstanding. The teacher still scores the target student's prefix, but it does so with evidence from local trial and error.

This does not require the teacher to be a different, larger model. In self-distillation, the teacher may have the same underlying model family but a richer information set. A model reading verified examples is solving a different conditional prediction problem from the same model reading only the task.

What a careful reader should expect

The book distinguishes four kinds of statements:

• Reported evidence: methods, settings, and measurements in the specified paper versions.
• Derivations: mathematical or numerical consequences, with assumptions stated.
• Interpretations: plausible explanations that may require further experiments.
• Proposals: implementation choices or extensions that the papers do not establish.

A reported improvement does not establish universal superiority. A low token-level divergence does not certify correctness. A teacher-student score gap does not by itself measure transfer difficulty. These distinctions become useful when the equations and benchmarks appear.

The goal is understanding rather than memorizing a recipe. By the end, a reader should be able to explain the objective, identify the assumptions behind a gradient estimator, calculate the headline metric, design a defensible experiment, and recognize where the evidence ends.

Questions to carry forward

What does a teacher know that a student does not? On which prefixes is that knowledge useful? How much of a teacher distribution must be transmitted? If specialists disagree, which behavior should the unified model learn? If a student never visits a successful branch, can distillation lead it there? These questions organize the rest of the book.

Source notes

The distinction between the two methods follows their titles, abstracts, and method sections. MT-MOPD's capability-production/capability-integration separation is developed in its Sections 3 and 5. MR-MOPD's peer-conditioned formulation appears in its Section 3. The examples and comparisons in this chapter are explanatory constructions.

Previous: Book introduction · Contents · Next: 2. Language Models as Probability Distributions and Policies

2. Language Models as Probability Distributions and Policies

On-policy distillation is easiest to understand when a language model is treated as a probability distribution over possible continuations. A response is not a single stored object. It is a sequence of decisions, each conditioned on the preceding history.

This chapter establishes the notation used throughout the book. It also explains why seemingly minor details such as tokenization, probability normalization, and loss masks can determine whether an implementation teaches the intended behavior.

Tokens, logits, and probabilities

A tokenizer converts text into integer token IDs. A token may represent a word, a word fragment, punctuation, or bytes. Let $V$ be the vocabulary. Given a context $c$, a model produces a real-valued logit $z_v(c)$ for every $v\in V$. Softmax turns these scores into probabilities:

The denominator matters. If a service returns only several logits, those logits alone do not determine their full-vocabulary probabilities. One also needs the full log-normalizer, or already normalized log-probabilities.

A token probability is conditional. Saying that a teacher gives “return” probability 0.2 is incomplete unless we say after which prompt and prefix. The same string can be reasonable after one history and nonsensical after another.

The models' vocabularies must also align for a direct tokenwise divergence. If token ID 123 means different text for two models, comparing their ID-123 probabilities is meaningless. A shared tokenizer and chat template are convenient compatibility conditions. Models with different tokenizations need another alignment procedure, which is outside the experiments considered here.

Autoregressive trajectories

Write a prompt as $x$ and a generated response as

The prefix before decision $t$ is $y_{<t}$. The student with trainable parameters $\theta$ defines

Its log-probability is a sum:

We often abbreviate the current context as $c_t=(x,y_{<t})$, the student's next-token distribution as $p_t$, and the teacher's distribution as $q_t$. Natural logarithms are used throughout; divergences are therefore measured in nats.

A trajectory includes the model's actual decisions. For a text-only task, those decisions are tokens. For a tool-using task, an interaction record also contains observations returned by the environment. Model-generated actions and environment-generated observations must be distinguished when computing a training loss.

Sampling creates the visited states

The model's probability distribution and its decoding rule are related but distinct. Greedy decoding selects a maximum-probability token. Sampling draws a token according to a distribution. Temperature modifies that distribution:

Top-$p$ or top-$k$ decoding additionally restricts support. A policy-gradient estimator assumes a relationship between the sampling distribution and the distribution whose log-probability is differentiated. If samples come from a truncated distribution but the estimator treats them as untruncated samples, that relationship changes.

The symbol $k$ has two unrelated uses in this book's subject. Top-$k$ decoding chooses what may be sampled. Top-$k$ distillation chooses which teacher probabilities enter a loss. Using 64 teacher tokens for distillation does not mean the student decodes with a 64-token sampling cutoff.

Teacher forcing is an evaluation operation

To score a student trajectory, a teacher reads the student tokens as given history. At position $t$, it predicts a distribution from $x,y_{<t}$ and evaluates the observed token $y_t$. It does not replace the student's earlier tokens with its own preferred choices.

For example, if the student's token sequence is 2 + 2 = 5, the teacher distribution at the final position is conditioned on 2 + 2 =. Its probability for 5 can be small even though the teacher would ordinarily generate 4.

A causal Transformer can evaluate many positions in parallel during a prefill pass because all input tokens are already known. Causal masking still prevents a position from seeing future target tokens. This distinction allows scoring to be faster than generating an equally long answer token by token.

The alignment is shifted by one token. The output after reading token $t-1$ predicts token $t$. A wrong shift can produce a loss that appears numerically valid while training the wrong targets.

A distribution contains more information than one sampled token

Suppose a student chooses action A. A scalar reward might say only that the full response failed. A sampled-token teacher signal might say that A is less likely under the teacher than under the student. A full distribution can also say which alternatives the teacher favors.

Consider this illustrative distribution:

The distributions are normalized. The teacher both discourages A and favors B. If only A is sampled, a sampled-token estimator observes A's correction on this visit; repeated visits can reveal other alternatives. Distribution-level distillation uses several alternatives immediately.

These are token probabilities in a toy vocabulary, not benchmark success probabilities. A high-probability token can belong to an ultimately incorrect trajectory. Local likelihood and end-to-end success are separate measurements.

Masks and length normalization

A real training record may include system instructions, user text, model reasoning, tool calls, tool outputs, and padding. Define a mask $m_t$ that is one for positions to train and zero otherwise. An average per-response loss can be written

assuming the denominator is nonzero.

This gives each response approximately equal weight. Averaging all eligible tokens across a batch instead gives longer responses more total influence. Neither choice is universally correct, but they represent different objectives.

For agents, an environment observation should usually serve as context rather than a target for the actor. Otherwise the model may be trained to imitate tool results it does not control. The papers' compact notation does not eliminate this implementation responsibility.

Three objects to keep separate

The student is the model being updated. The teacher supplies a conditional target distribution. The verifier evaluates task outcomes, such as whether a program passes tests. A teacher can be fluent and wrong; a verifier can be narrow or noisy. Their roles cannot be interchanged merely because both produce numbers.

MT-MOPD selects a domain teacher $q_d$. MR-MOPD additionally constructs a peer context $C_i$ and uses $q(\cdot\mid c_t,C_i)$. The student generally has no such extra context at deployment. That asymmetry is the learning problem, not a notation mistake.

Source notes

These definitions are mathematical background. The trajectory and context notation corresponds to the preliminary section of MR-MOPD and the shorthand around Equation 1 of MT-MOPD. The toy probabilities and masking discussion are educational examples and engineering analysis.

Previous: 1. Two Papers, One Acronym, Two Problems · Contents · Next: 3. Post-Training, Rewards, and Specialist Development

3. Post-Training, Rewards, and Specialist Development

Multi-Teacher MOPD is a post-training pipeline. Its teachers do not arrive as unexplained oracles: they are developed through domain-specific reinforcement learning from a shared supervised checkpoint. To understand the integration method, we first need to understand how those specialists obtain their skills.

What supervised fine-tuning teaches

A pretrained language model has learned broad statistical regularities from text. Supervised fine-tuning, or SFT, trains it on examples of desired responses. Given a target response $y^*$, the ordinary loss is

Every target token provides a learning signal. A well-designed corpus can teach conversation format, tool syntax, reasoning patterns, and instruction following. The model learns from supplied demonstrations rather than discovering responses by interacting with a reward function.

The training prefixes are the demonstration's prefixes. If the model produces a different prefix at deployment, its next-token behavior on that prefix may have received little direct supervision. This is one motivation for on-policy methods, though SFT remains useful for establishing a capable starting point.

MT-MOPD begins with broad SFT and uses the resulting checkpoint for both the student and all domain teachers. This shared origin is a deliberate part of the pipeline.

What reinforcement learning changes

Reinforcement learning lets the current model produce responses and adjusts their probability according to outcomes. In a simplified text-only setting,

A score-function gradient is

Subtracting a baseline that does not depend on the sampled action can reduce variance without changing the ideal expected gradient. Actual algorithms add stabilization, normalization, reference-policy penalties, and constraints on policy changes.

The challenge is that a single terminal reward may supervise thousands of decisions. A program can fail because of one missing check. Its trajectory-level reward alone does not tell the optimizer which token caused that failure.

Verifiable rewards and their limits

A math verifier can compare a parsed final answer to a reference. A coding verifier can execute tests. An instruction-following verifier can inspect constraints such as the number of list items or the absence of forbidden words. An agent task can evaluate whether the intended environment state was reached.

These are useful because they provide repeatable outcome signals. They are still measurements with scope. Passing a finite test suite does not prove that code is correct for all inputs. A final-answer check may accept an answer reached through an invalid derivation. A formatting check can miss semantic failures.

The relevant design question is what the reward actually measures. MT-MOPD depends on useful specialist behavior, so weaknesses in specialist rewards can become weaknesses in the integrated student. MR-MOPD additionally uses verifier outputs to label peers, making label quality directly affect the teacher context.

How rollout groups help GRPO

Group Relative Policy Optimization, GRPO, samples several responses for the same prompt and compares their rewards. A schematic standardized group advantage is

This expression captures the group-relative idea; it is not a complete specification of every GRPO implementation. The group provides a local baseline and scale. Responses better than their peers are reinforced relative to worse ones.

If all responses receive the same reward, the group may provide no relative learning signal. Dynamic sampling procedures can replace such groups with prompts that produce reward variation. This can improve the usefulness of a batch but also changes which prompts the optimizer sees.

MT-MOPD's specialist RL uses GRPO with Dynamic Sampling. Its integration stage does not use Dynamic Sampling and uses one rollout per prompt. Thus “multi-teacher” does not imply an eight-response group during integration. The eight-response groups belong to the reported specialist RL setup.

Why domains need different recipes

Math may have long responses and cheap answer checks. Repository repair may involve many tool turns and expensive execution. Instruction following may use shorter responses with multiple constraints. A good sampling strategy for one domain can be inefficient in another.

Reward distributions also differ. A batch of easy formatting tasks may contain many successes. A batch of difficult math tasks may contain almost none. Group standardization partly addresses scale but does not make all domains equivalent in difficulty, gradient variance, or compute cost.

Teacher development allows each domain to tune these choices independently. A coding team can change its execution environment without forcing the math team to restart. This is an organizational advantage as well as a learning design.

What interference means mathematically

Let $g_a$ and $g_b$ be gradients of two domain losses. Updating in the negative direction of $g_a$ changes the second loss approximately by

If the dot product is negative, an update that improves domain A locally can worsen domain B. If it is positive, the domains can help each other. Real models involve changing gradients and nonlinear trajectories, but this first-order calculation explains why independent skills do not automatically coexist.

Distillation can change the signals and improve their stability. It does not remove shared parameters. The final MT-MOPD integration still optimizes one student on a mixture of domains, so the possibility of interference remains.

The four alternatives MT-MOPD compares

Mix-RL pools domains in a joint RL run. It shares one student and one evolving optimization process.

Cascade RL trains domains sequentially. Its order can matter because each stage changes the initialization for the next.

Off-Policy Finetune trains specialists first, generates their responses, and fits a student to those static responses. It has dense supervised targets and independent teacher development, but its prefixes come from teachers.

Param-Merge combines specialist weights. It avoids a separate behavioral integration run, but its success depends on compatible parameter updates and a suitable merge recipe.

MT-MOPD uses independent specialists followed by behavioral integration on student trajectories. Its comparison is a comparison among complete recipes, not a theorem that the other families always fail.

Sparse and dense are descriptions of signals

A scalar reward is sparse in time: it often arrives at the end. A teacher distribution is dense: it gives a target at each trained token. Dense supervision can reduce uncertainty about useful local changes. It can also produce dense mistakes if the teacher is unreliable.

Sample efficiency therefore does not follow from density alone. It depends on the teacher's relevance, the student's visited states, the optimization method, and the cost of producing the target. Those dependencies become explicit in later chapters.

Source notes

The three-stage teacher-development setup and baseline definitions are reported in MT-MOPD, Sections 3–4 and Appendix A. The score-function and gradient-conflict calculations are background derivations. Details not specified by a paper, including a complete GRPO implementation, are not inferred from its short description.

Previous: 2. Language Models as Probability Distributions and Policies · Contents · Next: 4. Why the Student Must Practice Its Own Paths

4. Why the Student Must Practice Its Own Paths

Knowledge distillation transfers behavior from a teacher to a student. Its familiar form is imitation: collect teacher responses, then train the student to reproduce them. On-policy distillation changes the order. The student first produces a response; the teacher then supervises the distributions along that response.

The difference is about which states receive instruction. For an autoregressive model, a state is a prompt plus a partial response. Teaching a student on its own states makes the supervision responsive to its current behavior.

A branching example

Suppose a coding problem requires composing two mappings. A teacher's solution keeps the mappings separate from the beginning. The student instead assigns a variable a misleading meaning. At the fourth line, the student has entered a prefix that never occurs in the teacher's demonstrations.

Offline imitation repeatedly teaches the clean teacher prefix. It may eventually improve the student's early choice, but it does not necessarily teach how to continue or recover after the misleading assignment. On-policy supervision asks the teacher to read precisely that student prefix and score the next-token alternatives.

The distinction remains even if the teacher and student have the same architecture. It concerns sampled histories, not model size.

State distributions formalize the mismatch

Let $d_{\pi,t}(c)$ denote the distribution of contexts reached at position $t$ when policy $\pi$ generates responses. A teacher-generated corpus emphasizes $d_{q,t}$. Deployment with the student emphasizes $d_{\pi_\theta,t}$.

An offline objective might look like

An on-policy local objective instead uses

Two things changed in this illustrative pair: the context distribution and the KL direction. They are independent choices. One can use forward KL on student states or reverse KL on offline states. Calling a loss “on-policy” does not specify the divergence.

The benefit is that the teacher evaluates states likely under the current student. Some of those states contain errors. That does not make them worthless; they are exactly the states on which the student needs better decisions.

How errors can compound

Consider a stylized $T$-step process with a per-step probability $\epsilon$ of leaving the demonstration path. If steps were independent, the probability of remaining on that path would be $(1-\epsilon)^T$. With $\epsilon=0.01$ and $T=100$, that is roughly 0.366.

This calculation is illustrative rather than a claim about measured language-model errors. Real steps are dependent, and deviating from a demonstration can be harmless. It nevertheless shows why small local differences can create substantial changes in the distribution of long histories.

For an agent, an early action changes the environment. A different file edit produces different test output, which changes later context. The mismatch is then not simply a different sentence fragment: it can be a different world state.

What “no exposure bias” does and does not mean

MT-MOPD describes exposure bias as eliminated because it trains on student rollouts. The precise reading is that its integration targets are evaluated on prefixes generated by the current student rather than exclusively on teacher-generated prefixes.

This addresses a particular training-distribution mismatch. It does not ensure that deployment prompts match the training corpus. It does not ensure that deployment uses the same temperature, tools, token limits, or conversation format. The final student also differs from earlier students that generated training data.

On-policy sampling narrows a moving mismatch; it does not create permanent identity between every training and deployment distribution. Delayed asynchronous rollouts can also become stale as the optimizer advances.

A distillation step in ordinary language

An integration step can be understood as five operations:

1. Choose a task and, if relevant, its domain.
2. Let the student attempt the task under its current policy.
3. Feed the resulting history to a teacher using causal scoring.
4. Compare student and teacher next-token behavior on eligible positions.
5. Update the student and collect fresh attempts.

The teacher need not generate a free-running response. Its role is conditional evaluation. A prefill service can provide distributions along a known student sequence in one parallelizable pass.

MR-MOPD inserts another operation before teacher scoring: verify the rollout group and construct labeled peer context. MT-MOPD instead dispatches each trajectory to the appropriate domain specialist. The two changes affect different parts of the loop.

Why a poor attempt can still be useful

A scalar-reward method may obtain little relative information from a batch where every response fails. A teacher can still distinguish next-token alternatives within those responses. It may favor a missing check, a different tool, or a more plausible reasoning continuation.

There is a limit. If the entire prefix is irrecoverable, local next-token supervision may favor abandoning or correcting it, but cannot retroactively remove earlier tokens from that particular trajectory. Future rollouts can make different earlier decisions because their probabilities were also trained.

The method learns across attempts, not by magically repairing the already observed history. A sequence of local updates can move the policy toward better trajectories, but no general guarantee says it reaches every useful branch.

On-policy does not mean no replay of any kind

A strict fresh-rollout implementation uses data once and discards it. Other implementations perform several optimizer updates per sampled batch. The first update is closely aligned with the sampling policy; later updates use histories from an older policy.

Importance ratios and trust-region mechanisms can manage some consequences, but they do not erase the distinction. A report should specify rollout age, update count, and sampling distribution rather than rely on the phrase “on-policy” alone.

The multi-teacher paper reports single-use data for its specialist RL. Its appendix does not give a complete operational specification of every detail of integration updates. The reproduction chapter records that boundary explicitly.

Why this is still distillation rather than ordinary RL

The target comes from a teacher's predictive distribution. That target may reflect successful RL training, but a token-level likelihood difference is not a directly measured environment reward. A model can regard a familiar wrong continuation as likely.

On-policy distillation borrows sampling and policy-gradient machinery from RL while optimizing teacher agreement. Outcome evaluation remains essential to verify that agreement produces useful behavior.

This distinction also explains why teacher selection matters. The student should agree with a teacher on the relevant task, not indiscriminately agree with every available model.

Source notes

The on-policy definitions are grounded in MR-MOPD, Section 2 and MT-MOPD, Section 3. The state-distribution formulation, compounding example, and qualifications of exposure-bias language are explanatory analysis.

Previous: 3. Post-Training, Rewards, and Specialist Development · Contents · Next: 5. What KL and Jensen–Shannon Actually Teach

5. What KL and Jensen–Shannon Actually Teach

A distillation loss compares two probability distributions. The choice of comparison changes which discrepancies receive the most pressure. To read MOPD's equations accurately, we need more than the phrase “match the teacher.” We need to know which distribution supplies the expectation and how the loss responds to rare events.

Throughout this chapter, $p$ is a student distribution and $q$ is a teacher distribution at one fixed context. All probabilities are over the same vocabulary, and logarithms are natural.

Forward KL

The teacher-to-student direction is

Because $q$ is fixed, minimizing this is equivalent to minimizing teacher-weighted cross-entropy:

If the teacher gives an alternative substantial mass while the student gives it almost none, the penalty is large. The teacher's weighting draws attention to alternatives the student might rarely sample.

This is why forward KL is often described as mode covering. Under a restricted student family, it tends to avoid abandoning teacher-supported possibilities. With an unrestricted probability vector, however, its optimum is simply $p=q$. It is not intrinsically an instruction to generate diverse outputs regardless of the teacher.

Reverse KL

The student-to-teacher direction is

Now the student supplies the weighting. If it puts substantial probability on an option that the teacher considers very unlikely, the penalty is large. If it barely samples an option the teacher likes, that option initially receives little direct sampled-token attention.

Rewrite the expression as

where $H(p)=-\sum_vp_v\log p_v$ is entropy. Minimizing reverse KL balances teacher likelihood with student entropy. It is not equivalent to maximizing teacher likelihood alone.

Calling reverse KL mode seeking describes a tendency when a limited family approximates a multimodal target, or when optimization and sampling restrict access to modes. If the student can represent the teacher distribution exactly and optimization succeeds, it can match all modes. The slogan is a useful warning, not a universal collapse theorem.

Numerical comparison

Take two possible next tokens with

The reverse KL is approximately

The forward KL is approximately

Both favor moving the student toward $(0.5,0.5)$, but they assign different costs to its undercoverage of the second token.

Now consider a sampled first token. Its teacher-student log difference is

For the second token, the difference is $\log 5\approx1.6094$. The second token would receive a positive signal if sampled, but it appears only 10% of the time under the current student. This connects divergence geometry to estimator variance and exploration.

Support and numerical zeros

If $q_v=0$ while $p_v>0$, reverse KL is infinite. If $p_v=0$ while $q_v>0$, forward KL is infinite. Standard softmax with finite logits gives positive probabilities in exact arithmetic, but floating-point underflow, truncation, and sparse payloads can create effective zeros.

A log-probability floor can prevent numerical infinities, but it changes the target. Renormalizing a top-$k$ teacher distribution also changes the target. These choices should be documented because they affect what “KL” means in an implementation.

Use log-softmax and log-domain differences where possible. Subtracting normalized log-probabilities is more stable than taking logarithms of low-precision probabilities after exponentiation.

Jensen–Shannon divergence

Define the mixture $m=(p+q)/2$. Jensen–Shannon divergence is

It is symmetric and lies between zero and $\log 2$ for normalized distributions. It remains finite when the two distributions have disjoint support. These properties can make it less aggressive than a one-sided KL in difficult teacher-student alignments.

The mixture depends on both distributions. Correct automatic differentiation must account for that dependence unless one deliberately uses a surrogate. Simply combining two arbitrary KL calculations with detached mixture probabilities need not produce the JS gradient.

MR-MOPD reports reverse KL for math and JS for the other tasks. MT-MOPD uses reverse KL and gives sampled-token and corrected top-$k$ implementations. These are different reported choices, so a single “MOPD loss” does not describe both papers.

Temperature is part of the target

If logits are divided by temperature before normalization, the target distribution changes. A higher temperature usually reveals more mass in lower-ranked alternatives. A lower temperature concentrates mass near the top.

Classical distillation sometimes introduces temperature and scales the loss to adjust gradient magnitude. That is a possible design choice, not something to silently assume for these experiments. The papers' evaluation temperature is also distinct from any temperature used in constructing teacher targets.

When comparing methods, track sampling temperature, scoring distribution, and deployment decoding separately.

Divergence, correctness, and diversity are different

A student can reach zero divergence from a wrong teacher. A correct student can differ from a teacher because it expresses another valid solution. A teacher can have low entropy yet make a consistent mistake. No divergence value alone establishes task success.

Likewise, token entropy is not the same as solution diversity. Different lexical choices can leave an algorithm unchanged. A small change in one control-flow token can produce a different algorithm while barely changing average entropy.

This is why MOPD evaluation needs both task outcomes and distributional diagnostics. KL describes agreement; a verifier describes a specified success condition; diversity metrics describe variation under their chosen representation.

Choosing a loss in practice

For compatible same-origin teachers, reverse KL can offer direct supervision along student behavior. For a larger distributional mismatch, merely changing to JS may help bound penalties but cannot guarantee successful transfer. Support, capacity, data coverage, and learning rates still matter.

The right question is which discrepancy a method must fix and whether its estimator sees that discrepancy. A teacher-weighted loss can expose alternatives the student seldom samples. A student-weighted loss emphasizes its actual mistakes. A sparse approximation trades information for cost.

Source notes

The divergence definitions and MR task choices appear in MR-MOPD, Section 3 and its divergence appendix. MT's reverse-KL implementations appear in MT-MOPD, Section 3.2. Numerical calculations and the discussion of support, normalization, and temperature are derivations and implementation analysis.

Previous: 4. Why the Student Must Practice Its Own Paths · Contents · Next: 6. The Multi-Teacher Pipeline from Beginning to End

6. The Multi-Teacher Pipeline from Beginning to End

Multi-Teacher MOPD separates producing capabilities from integrating capabilities. The distinction explains both its algorithm and its proposed workflow advantage. Specialists learn through domain-specific RL; a unified student then learns from those specialists on its own trajectories.

This chapter describes the reported pipeline before later chapters examine its objective and engineering details.

Stage 1: establish a shared supervised origin

Begin with a base model and perform broad supervised fine-tuning. Call the resulting parameters $\theta_0$. This checkpoint should cover the response formats and basic capabilities needed across target domains.

The student begins integration at $\theta_0$. Every domain teacher also begins its specialist RL at $\theta_0$. A specialist is therefore an adaptation of a common origin rather than an unrelated model chosen only for its benchmark score.

The paper's controlled experiments use Qwen3-30B-A3B-Base and a custom SFT pass. Its conclusions apply to that experimental starting point. They are not measurements of the standard public instruct checkpoint under an unspecified recipe.

Stage 2: develop domain specialists independently

Let $d\in\{1,\ldots,D\}$ index domains. Specialist RL produces teacher parameters $\phi_d$ from $\theta_0$:

The expression summarizes a training procedure rather than a closed-form equation. Each domain has its own prompts, reward, rollout procedure, and hyperparameters.

The controlled study uses math, instruction following, and software engineering. The larger MiMo application additionally includes code and tool use. These domain sets should not be conflated.

Teacher development can run concurrently if resources permit. Independence means the math optimizer does not update the software teacher's weights. It does not mean the teachers have zero operational dependencies: they may still share data infrastructure, compute allocation, evaluation tools, and the original checkpoint.

Stage 3: freeze teachers and integrate behavior

During one integration round, freeze the teacher set. Initialize the student from the Stage-1 checkpoint and draw mixed-domain prompts. For each prompt, the student generates a trajectory. The task's domain dispatches that trajectory to its specialist teacher.

The teacher reads the student's history and returns tokenwise probability information. The optimizer then updates the student using a reverse-KL-based loss.

A compact schematic is:

                     broad SFT checkpoint
                       /      |      \
                  math RL    IF RL    SWE RL
                     |         |         |
                 teacher M  teacher I  teacher S
                       \       |       /
                   domain-routed supervision
                              |
             student rollouts from the SFT initialization
                              |
                       one integrated model

The diagram separates training roles. It does not depict three teacher models being loaded inside the final inference graph.

What exactly is routed

The training task carries a domain label $d(x)$. The target is

The reported design routes each prompt to its corresponding domain teacher. It does not average all teacher distributions on every token. It does not introduce a learned tokenwise mixture-of-experts router. It does not establish a universal scheme for prompts that simultaneously require several domains.

These distinctions matter because averaging distributions defines a different objective. If the math and instruction-following teachers disagree on response style, a single selected teacher and a probability mixture apply different training pressure.

Why the teachers can be frozen

A frozen teacher provides a stable target during integration. The student changes, so the prefixes on which the teacher is queried change, but the teacher's parameters remain fixed.

This still creates a changing curriculum. A teacher can assign different predictions to different student prefixes without being updated. The targets adapt to instances through conditioning, not through teacher parameter training inside the integration pass.

Later rounds can develop new teachers from the integrated student. That creates a new stage with a new target set. It should not be described as teachers continuously co-training with the student within one round.

A complete conceptual iteration

For each sampled training record, retain its prompt, domain, rollout policy version, generated token IDs, masks, and behavior-policy log-probabilities. Query the selected teacher with compatible formatting and the observed trajectory. Receive sampled-token log-probabilities or a top-$k$ payload. Recompute student log-probabilities for optimization and apply the selected loss.

After an update, fresh rollouts reflect the new student. Previously common errors may become rarer, while new prefixes become common. The teacher's conditional scoring follows this moving distribution.

The paper presents two ways to implement the local objective. A policy-gradient form uses the log difference for the sampled token. A top-$k$ form uses a corrected partial-distribution divergence. They share the pipeline but do not have identical finite-sample gradients.

What the modular design buys

A successful specialist can be retained while another domain is retrained. Domain teams can explore different RL recipes. An unstable specialist run can be restarted without restarting the others. These advantages arise before integration begins.

The integration stage still needs a shared sampling mixture and optimizer. A weak domain teacher can inject poor targets. A new teacher may change the balance of student updates. The architecture organizes coupling; it does not abolish all coupling.

A useful operational unit is a teacher bundle: checkpoint identity, tokenizer, chat template, scoring endpoint, domain eligibility, evaluation evidence, and target-distribution settings. Replacing only a model filename without checking the other pieces can invalidate supervision.

What the final student contains

The final student has one parameter set. Its different capabilities are represented through shared weights and conditional behavior. The teacher pool is a training resource, not necessarily a deployment requirement.

This can reduce serving complexity compared with an inference-time ensemble. It does not establish model compression in parameter count. In the main study, same-origin specialists and student have the same architecture. The method's main purpose is capability consolidation.

Smaller students and heterogeneous teachers are possible distillation scenarios, but the paper's strongest evidence concerns its same-origin specialist recipe. Generalizing beyond it needs additional validation.

Reported configuration and missing details

For the controlled MT integration, the appendix reports batch size 2,048, one rollout per prompt, domain proportions math:IF:SWE of 0.35:0.35:0.30, no Dynamic Sampling, advantage clip maximum 5, and top-$k$ size 64 for that alternative loss.

It does not provide a complete public end-to-end training specification. A reader should not fill missing integration learning-rate, optimizer, serving, or masking details by assuming they match specialist RL. The implementation blueprint later labels such choices explicitly.

Source notes

The stages, routing, frozen teachers, and service organization are reported in MT-MOPD, Section 3. Hyperparameters are from Appendix A. The teacher-bundle abstraction and record schema are implementation recommendations, not claimed paper features.

Previous: 5. What KL and Jensen–Shannon Actually Teach · Contents · Next: 7. The Reverse-KL Objective and Its Gradient

7. The Reverse-KL Objective and Its Gradient

Multi-Teacher MOPD uses reverse KL between the student and the selected domain teacher at each prefix of a student-generated response. This chapter derives what that objective means and addresses a subtle issue: the prefixes are themselves generated by the parameters being optimized.

The local gradient is straightforward. The gradient of a complete on-policy expectation needs additional care. Keeping those two objects separate prevents a common misunderstanding of policy-gradient distillation.

Write the complete objective first

Let $x$ be a prompt, $d(x)$ its domain, and $y\sim\pi_\theta(\cdot\mid x)$ a student trajectory of length $T$. Define

The paper's Equation 1 is

The vocabulary sum compares all possible next tokens at a visited prefix. The trajectory average combines prefixes. The outer expectation combines prompts and generated responses. Each layer is a distinct source of weighting.

The teacher is frozen in parameters. Its output still depends on the prefix. If the student's earlier choices change, the teacher's conditional prediction can change as well.

Fix one context and differentiate

At a fixed context $c$, define

Differentiate with respect to $\theta$, holding $c$ and the teacher distribution fixed:

Because the student probabilities sum to one,

The constant term cancels. Using $\nabla p_v=p_v\nabla\log p_v$ gives

Equivalently,

This is the mathematical basis for the teacher-student log-difference signal.

Why the log difference acts like an advantage

Define

If the teacher assigns more probability than the student, $A$ is positive. If it assigns less, $A$ is negative. Minimizing a surrogate $-A\log p$ increases the sampled token's probability for positive $A$ and decreases it for negative $A$, subject to the coupling created by softmax.

The word “advantage” describes the estimator's role. This quantity is not the same as a measured task advantage or an expected return-to-go. It measures relative conditional likelihood.

For $p_v=0.1,q_v=0.4$, $A=\log4\approx1.3863$. For $p_v=0.4,q_v=0.1$, $A=-1.3863$. Equal probabilities give zero signal for that token.

A state-only baseline $b(c)$ can be subtracted without changing the ideal local expectation because $\mathbb E_{v\sim p}[\nabla\log p_v]=0$. In practice, clipping, estimated baselines, and stale samples complicate that statement.

Stop-gradient is part of the estimator

If we implement

automatic differentiation produces $-A\nabla\log p$. The symbol $\operatorname{sg}$ means that $A$ is treated as a constant during this update.

If the student term inside $A$ is left attached, differentiation adds another term. The result is a different surrogate gradient. Detaching the advantage is therefore an algorithmic requirement for the stated estimator, not just a memory optimization.

The surrogate's scalar value is not KL. It can be negative and need not vanish when an estimated KL would. Monitor the objective and the training surrogate separately.

The outer trajectory expectation is also parameter-dependent

Now return to the complete objective. Let

For a fixed prompt, the exact score-function identity is

under the usual differentiability and summability conditions. The first term differentiates local losses on an observed trajectory. The second term accounts for changes in which trajectories are visited.

A procedure that collects trajectories and then treats their prefixes as fixed computes a local or semi-gradient update. It generally omits the occupancy-distribution term. Recollecting trajectories after updates makes the procedure on-policy operationally, but does not make the omitted term algebraically disappear.

The MT paper presents its Equation 2 as the gradient of Equation 1. Its immediate sampled-token expression is naturally justified by the fixed-context derivation above. The book does not treat that short expression as a complete proof of the total derivative through all student-induced state visitation.

Sequence KL is another objective

For fixed-length trajectories with common support, a sequence-level reverse KL is

The chain rule relates its value to expected conditional KLs. But its full policy gradient includes effects of early actions on later log-ratio costs. A reward-to-go formulation can assign future costs to earlier decisions.

The paper's immediate token advantage does not explicitly contain those future terms. Length normalization by a random $T$ adds further weighting differences. A variable-length, per-response average is not automatically identical to unnormalized sequence KL.

These distinctions do not invalidate an effective local training recipe. They identify which mathematical claim requires which assumptions.

A two-step thought experiment

Suppose a student's first token chooses branch A or B. On both branches, its first-token distribution is close to the teacher's. At the second token, branch A creates a large mismatch while branch B creates none.

A local first-token gradient sees only the first-token mismatch. A full trajectory-distribution gradient also sees that increasing the probability of branch A increases the chance of visiting an expensive second-step state.

The two gradients can differ even though both algorithms sample trajectories from the current student. This example isolates the role of visitation without relying on complicated model internals.

In real reasoning tasks, an early decision can place the model in a prefix where the teacher has poor recovery behavior. Whether optimizing occupancy improves learning is an empirical and algorithmic question, not something settled by the term “on-policy.”

What can be claimed precisely

The sampled log-difference gives an unbiased estimator of a fixed-context reverse-KL gradient when sampled from the current student distribution, with no clipping and the relevant context weighting held fixed. Using freshly sampled contexts targets the student's current experience in a local-update sense.

Clipping creates a stabilized surrogate. Top-$k$ creates a partial-support objective. Repeated updates on old trajectories introduce sampling mismatch. These are reasonable modifications, but each changes the exactness statement.

A careful implementation documents the intended objective, estimator, approximations, and monitoring quantities independently. That clarity is more useful than describing every component as “exact reverse KL.”

Source notes

The displayed MT objective, log-difference advantage, and stop-gradient notation correspond to MT-MOPD, Equations 1–4. The fixed-context derivation and occupancy-gradient qualification are independent mathematical analysis. They are not additional results claimed by the paper.

Previous: 6. The Multi-Teacher Pipeline from Beginning to End · Contents · Next: 8. Implementing the Sampled-Token Update

8. Implementing the Sampled-Token Update

The policy-gradient form of Multi-Teacher MOPD reduces the teacher payload to one log-probability for each sampled student token. Its economy is attractive: a rollout already contains the chosen tokens, and the teacher only needs to report how likely it considers those tokens at the same prefixes.

The implementation must preserve the distinction between the policy that sampled a trajectory, the policy being updated, and the frozen teacher. This chapter turns the formula into an auditable training step.

The three probabilities at one position

Let $p_b$ denote the behavior student that generated the rollout. Let $p_\theta$ denote the student evaluated for the optimizer update. Let $q_d$ denote the routed teacher.

In a strictly fresh one-update setting, $p_b$ and the pre-update $p_\theta$ coincide. In asynchronous or repeated-update settings, they can differ. Recording the behavior log-probability makes that difference visible.

The reported immediate advantage has the form

The precise student snapshot used to construct that detached quantity is an implementation detail that should be explicit. Using rollout-time probabilities and using newly recomputed probabilities are not interchangeable once the model changes.

Two-sided advantage clipping

The MT paper clips the log difference:

It then minimizes

The default reported clip maximum is 5. A magnitude of 5 corresponds to a likelihood ratio $e^5\approx148.4$. More extreme ratios receive the same bounded advantage magnitude.

This is advantage clipping. PPO's probability-ratio clipping is another mechanism with a different mathematical purpose. Reusing PPO software does not make the two clipping operations equivalent.

Clipping lowers the influence of outliers but biases the local reverse-KL gradient. That tradeoff is intentional stabilization. Report the fraction of tokens clipped, preferably separately for positive and negative values, because heavy clipping can obscure a distributional mismatch.

Follow the sign through an update

For a sampled token with positive $A$, the loss contribution is negative $A\log p$. Gradient descent tends to increase its log-probability. For a negative $A$, the update tends to decrease it.

Softmax couples tokens: changing one logit also changes the probabilities of others. A negative update does not specify a single replacement token. The remaining mass redistributes through the parameterized network and future examples.

A toy example makes the weighting concrete:

The method provides different local pressures even if all three tokens belong to one trajectory with a single terminal failure label. The toy probabilities are explanatory, not measured outputs from the papers.

Conceptual pseudocode

The following illustrates a fresh, one-update local surrogate. It is not a reproduction of an undisclosed training repository.

batch = sample_domain_labeled_prompts(domain_mixture)
rollouts = student_sampler.generate(batch, policy_version=current_version)
records = attach_masks_and_behavior_logprobs(rollouts)

teacher_logp = score_sampled_tokens_by_domain(records, frozen_teachers)
student_logp = actor.logprobs(records.token_ids, records.attention_mask)

# This conceptual version assumes behavior and pre-update actor coincide.
advantage = stop_gradient(teacher_logp - records.behavior_logp)
advantage = clamp(advantage, -5.0, 5.0)

per_token_loss = -advantage * student_logp
per_response_loss = masked_response_mean(per_token_loss, records.actor_mask)
loss = mean(per_response_loss)
loss.backward()
optimizer.step()

For stale records, choose and document an importance-weighted or otherwise controlled update rather than silently applying the same exactness claim. An importance ratio $p_\theta(y_t\mid c_t)/p_b(y_t\mid c_t)$ addresses token sampling at a fixed context; it does not by itself correct the entire prefix distribution.

Why the teacher service should return log-probabilities

A raw logit can shift by an arbitrary context-dependent constant without changing the distribution. A logit difference between teacher and student therefore is not a probability log-ratio.

The target service should return normalized log-probabilities for the sampled IDs. If it returns logits, it must also return sufficient information to recover the full log-normalizer.

This requirement is easy to miss when adapting a generation endpoint. An endpoint that returns only its own selected tokens' scores may not support scoring arbitrary student tokens. A valid distillation endpoint needs causal teacher forcing over the supplied sequence.

Which tokens receive loss

Trainable positions normally include student-generated reasoning, answers, and action tokens according to the chosen recipe. Prompts, padding, and environment observations are conditioning information rather than actor decisions.

For a multi-turn agent, masks must track message roles. A tool output containing a stack trace can be hundreds of tokens long; training it as model output would distort weighting and teach behavior the actor does not control.

Length normalization also matters. The paper writes a per-trajectory average. A batch-level token mean would favor long agent histories. Both can be implemented, but only one matches that displayed weighting.

What to monitor

Track task success, mean and quantiles of teacher-student log differences, clipping rates, entropy, response length, gradient norms, and per-domain losses. Track a separately estimated divergence if desired; the signed surrogate loss is not a nonnegative KL value.

Break diagnostics down by domain and by trajectory outcome. An improving aggregate loss can hide deterioration on a small but important domain. A shortening response distribution may improve throughput while destroying hard reasoning.

When instability appears, first check token alignment, normalization, masks, and stale-policy handling. Algorithmic explanations are premature if the service is scoring the wrong token at the wrong position.

Why only one sample can be sufficient locally

A vocabulary expectation can be estimated with one sampled token. That statement concerns unbiasedness under the local assumptions, not low variance. Over a large batch, many token samples can collectively provide a useful gradient.

Same-origin teachers help when the student's sampled tokens lie in regions where teacher probabilities are reasonably aligned. If most sampled tokens receive strongly negative log differences, a single-token estimator may become punitive and narrow the policy. A stronger teacher can therefore be less useful than a closer one under a fixed recipe.

Source notes

The advantage, clip, and sampled-token loss are reported in MT-MOPD, Section 3.2.1 and Appendix A. Behavior-policy distinctions, pseudocode, masking rules, and monitoring suggestions are implementation analysis. The paper's statement that the form fits existing PPO/GRPO infrastructure should not be read as a full specification of stale-rollout corrections.

Previous: 7. The Reverse-KL Objective and Its Gradient · Contents · Next: 9. Why Top-k Distillation Needs a Correction

9. Why Top-k Distillation Needs a Correction

A sampled-token update uses one teacher probability per position. Full-vocabulary distillation uses every probability, which can be expensive to transfer and store. Multi-Teacher MOPD offers a middle ground: score the teacher's top-$k$ tokens at each student prefix.

Restricting a reverse-KL sum to a subset of tokens is not innocuous. The ordinary cancellation that makes the KL optimum occur at equality depends on including all normalized probabilities. The paper adds a correction term to restore the intended optimum on the selected coordinates.

The reported sparse objective

At one fixed context, let $S=\operatorname{TopK}_k(q)$ be the teacher's top-$k$ token set. The paper uses

The probabilities retain their full-vocabulary normalization. They are not renormalized to sum to one over $S$. The default reported $k$ is 64.

This loss compares selected probability masses. It does not assert that all probability mass belongs to the selected tokens.

Why a naive restricted KL is wrong

Suppose we keep only

For one selected coordinate, treating $p_v$ as a free positive scalar gives

Its stationary point is $p_v=q_v/e$, not $p_v=q_v$. In the full vocabulary, normalization constraints cancel the constant terms jointly. On a subset, mass can move to unscored tokens instead.

Take a two-token example with teacher $q=(0.8,0.2)$ and score only the first token. The naive loss at equality is zero. At $p_1=0.8/e\approx0.2943$, it is approximately $-0.2943$, with the remaining student mass 0.7057 assigned to the unscored token. The naive objective prefers this mismatched distribution.

Thus a negative “partial KL” value is not evidence of excellent teacher matching. It may be a symptom of the wrong sparse objective.

The correction restores coordinate equality

For the corrected term,

we have

The unique coordinatewise minimum is at $p_v=q_v$, where the term is zero. To see nonnegativity, write $r=p_v/q_v$:

The scalar function is convex and minimized at $r=1$.

In the toy example, the corrected loss at $p_1=0.2943$ is approximately

whereas it is zero at $p_1=0.8$. The added term reverses the spurious incentive to evacuate selected probability mass.

What “bias correction” means here

The correction fixes the selected-coordinate optimum. It does not make top-$k$ truncation an unbiased estimator of the full-vocabulary reverse-KL gradient. Unselected terms are still absent.

If all selected student probabilities match the teacher, the selected loss is zero. Normalization then forces the total unselected mass to match as well, but it does not determine how that mass is allocated among tail tokens.

For example, let

and score only the first token. The corrected loss is zero even though the tail probabilities differ. The full reverse KL is $0.1\log0.5+0.2\log2\approx0.0693$.

A missing tail can contain important rare actions. Top-$k$ is an information-budget choice rather than a guarantee that low-ranked tokens do not matter.

Gradients through the student softmax

The student probabilities share a normalization. Let $p=\operatorname{softmax}(z)$ and define

Then

Even unselected logits can receive a gradient through the normalization term. The loss ignores their individual teacher probabilities, but changing selected mass affects the whole distribution.

This is why computing student probabilities with softmax only over selected logits is not equivalent. That operation creates a new normalized subdistribution and a different gradient.

Numerical implementation

For selected IDs, obtain teacher log-probabilities $\log q_v$ normalized over the full teacher vocabulary. Compute student log-probabilities $\log p_v$ from the full student logits, then gather the selected entries. The selected loss can be evaluated as

The final teacher term is constant with respect to student parameters. Dropping it would preserve the student gradient but destroy the nonnegative, zero-at-equality interpretation of the logged scalar loss. Keeping it helps diagnostics.

A practical implementation may compute normalization in higher precision than stored model weights. Confirm that probabilities near the numerical floor do not produce NaNs or unexpectedly large ratios.

Is the sampled token always in the payload?

No. The student can sample a token outside the teacher's top-$k$. The corrected sparse objective still scores the teacher's selected alternatives at that prefix. It does not automatically provide that sampled token's teacher log-probability.

If an implementation also wants sampled-token diagnostics or a hybrid update, request that probability separately. Adding the sampled token to the scored set is a possible extension, but it changes the exact set used by the reported top-$k$ objective and should be labeled accordingly.

A top-$k$ endpoint should return IDs and normalized values at every eligible position, not merely the teacher's preferred next word once for the whole response.

Information and cost tradeoffs

The sparse payload carries $k$ probabilities and $k$ IDs per position. For $k=64$, 32-bit IDs, and 32-bit log-probabilities, that is 512 bytes per position before metadata. A full distribution over a hypothetical 150,000-token vocabulary at four bytes per value is 600,000 bytes per position.

This reduces transmission substantially. It does not reduce the teacher's entire forward pass by the same factor: the teacher may still compute a full output head before selecting its top tokens. Gathering and selection have their own cost.

Teacher mass captured by top-$k$ should be monitored. A fixed $k$ may capture almost all mass in one context and little in another. Loss comparisons across domains can therefore reflect different coverage.

What the experiments establish

The same-origin comparison reports normalized scores of 0.9373 for policy gradient and 0.9093 for top-$k$. The paper interprets them as broadly comparable, with no additional stability benefit from top-$k$ in that setup. Under its external-math-teacher replacement, top-$k$ performs dramatically worse.

Those observations show that additional distributional information does not automatically produce a better result under a fixed recipe. They do not establish that sparse distillation is universally worse or that increasing $k$ cannot help. Learning rate, support mismatch, and target compatibility remain relevant.

Source notes

The objective and correction motivation are from MT-MOPD, Equation 5 and Section 3.2.2. The coordinatewise and logit derivatives, tail counterexample, and payload arithmetic are independent derivations. Comparisons are from Table 4.

Previous: 8. Implementing the Sampled-Token Update · Contents · Next: 10. Routing, Teacher Disagreement, and Shared Capacity

10. Routing, Teacher Disagreement, and Shared Capacity

The “multi” in Multi-Teacher MOPD does not mean all teachers vote on every token. Each training task is routed to the corresponding specialist. This apparently simple design choice determines which conflicts the student must resolve and which conflicts are left outside the method's scope.

This chapter examines routing as an objective-design decision rather than a serving detail.

The domain mixture defines priorities

Let $w_d$ be the probability of sampling domain $d$, and let $\mathcal D_d$ be its prompt distribution. A local multi-domain objective can be written

The weights express how often each domain supplies training pressure. They are not a measured statement that domains have equal difficulty or equal societal value.

The reported controlled mixture is 0.35 math, 0.35 instruction following, and 0.30 software engineering. Its headline evaluation score, by contrast, weights the three domains uniformly. Training weights and evaluation weights have different roles.

Prompt proportions also do not equal compute proportions. A software trajectory can contain far more tokens and costly tool interactions than an instruction-following response. Per-response normalization partly changes gradient weighting but does not change how long the trajectory takes to produce.

Routed supervision is a conditional target

For a math task, the target is the math teacher. For an instruction-following task, the target is the instruction-following teacher. The student sees the prompt, not necessarily an explicit teacher label in its deployment input.

In principle, the model can learn different conditional behaviors because different prompts indicate different tasks. Shared parameters still represent those behaviors. A finite-capacity student may have to compromise when domain-specific updates affect common representations.

The method does not assume that every teacher is best on every task. A repository specialist might be an unnecessarily verbose teacher for a short formatting task. Routing limits that source of irrelevant supervision.

What if all teachers are averaged?

Suppose several teachers provide distributions $q_d$ on the same context and we minimize a weighted sum of reverse KLs:

Ignoring student-family restrictions, the minimizing distribution is proportional to a weighted geometric mean:

To derive this, expand the objective into $\sum_vp_v\log p_v-\sum_vp_v\sum_dw_d\log q_d(v)$ and impose normalization with a Lagrange multiplier.

This objective can suppress options strongly rejected by any teacher. It differs from distilling an arithmetic probability mixture $\sum_dw_dq_d$. It also differs from the reported per-prompt routing scheme.

The distinction matters when teachers have different styles or tool protocols. Combining contradictory targets at one state is an algorithmic change, not merely a different deployment of the same method.

Mixed-domain requests expose a boundary

A user might request a mathematical solution in an exact JSON format, or a software patch plus a constrained explanation. Such a request spans domains. A prompt-level label chooses one teacher unless a more elaborate scheme is introduced.

The paper does not supply a complete routing theory for these cases. It demonstrates integration across domain-labeled evaluation families. A student's success across those families does not automatically establish competence at every cross-domain combination.

Possible extensions include compound-task datasets, a specialist trained for combined constraints, or a validated arbitration rule. Tokenwise routing might separate reasoning and formatting, but it raises new questions about coherent targets and message boundaries. These are proposals, not reported MT-MOPD features.

Why shared origin helps routing

Specialists from one SFT checkpoint often share tokenization, chat formatting, and broad response conventions. Their target distributions may therefore differ mainly in domain skill rather than every aspect of style.

This reduces avoidable disagreement. It does not ensure identical formatting or low divergence after arbitrarily long specialist RL. A teacher that learns a distinctive response style can still create conflicting pressure on shared student parameters.

Compatibility should be measured at student-generated contexts. High agreement on an unrelated offline corpus can miss the states actually visited during integration.

Integration still has gradient conflict

If domain losses have gradients $g_d$, the mixed update follows $\sum_dw_dg_d$. A teacher's dense signal can improve local information while still conflicting with another domain's signal.

The observed balanced performance in the controlled experiment is evidence that this recipe handled its domains effectively. It is not proof that distillation removes the capacity and optimization challenges of multi-task learning.

Useful diagnostics include per-domain gradient norms, per-domain outcome curves, and regression tests for general SFT abilities. A normalized aggregate score can improve even while one capability deteriorates.

Teacher replacement is a controlled change

Replacing a teacher changes the target distribution for one domain. It may also change payload size, numerical behavior, response conventions, and latency. Keep those variables visible when interpreting the outcome.

A practical replacement evaluation should compare benchmark competence, initial divergence on current student prefixes, clipping rates, support coverage, and stability under a short integration run. Higher teacher accuracy is relevant but insufficient.

The paper's external-math-teacher experiment illustrates this: a stronger math model supplies a less compatible target under the tested recipe. The following chapter examines the evidence in detail.

A defensible routing contract

Define which tasks a teacher is allowed to supervise, how domain labels are assigned, and how ambiguous tasks are handled. Record the chosen teacher identity with each training example. Make routing deterministic enough to audit before exploring adaptive alternatives.

At deployment, evaluate combined tasks explicitly. Capability integration should be measured through both separate-domain tests and conjunction tests that require abilities to coexist in one response.

Source notes

Prompt routing and the controlled mixture are reported in MT-MOPD, Section 3 and Appendix A. The geometric-mean derivation, mixed-domain analysis, and routing contract are independent analysis. They describe alternatives and boundaries rather than undocumented paper mechanisms.

Previous: 9. Why Top-k Distillation Needs a Correction · Contents · Next: 11. Why a Stronger Teacher Can Teach Worse

11. Why a Stronger Teacher Can Teach Worse

A teacher's benchmark score measures how well it performs a task under an evaluation protocol. It does not directly measure how well its conditional distributions supervise a particular student's trajectories. Multi-Teacher MOPD's stability analysis makes this distinction unusually concrete.

The paper replaces its math specialist with a larger, stronger model and observes worse student performance. Understanding that result requires separating teacher competence from teacher-student compatibility.

What same-origin means

The default domain teachers are RL adaptations of the SFT checkpoint that initializes the student. They share architecture and a recent parameter origin. The paper argues that this produces a smaller initial policy divergence and smoother integration.

Same origin is a practical way to encourage alignment. It is not a mathematical guarantee. A specialist can move far from its origin, and two same-origin models can differ strongly on particular contexts.

The operative quantity is the target distribution on the student's visited prefixes. A model with another origin might be compatible after adaptation; a same-origin model might become incompatible after aggressive specialization.

The reported replacement experiment

In the controlled three-domain setup, the authors replace the Qwen3-30B-A3B math RL teacher with Qwen3-235B-A22B. IF and SWE teachers, data, and the remaining setup are held fixed according to the paper.

These are the paper's Table 4 values. Benchmark columns are percentages. The normalized score uses the original study's student/teacher reference anchors; it is not a direct score for the external teacher.

The catastrophic top-$k$ math results deserve attention, but they are one recipe's outcome. They do not establish that Qwen3-235B-A22B cannot be used as a distillation teacher under another setup.

Reported distributional diagnostics

The paper describes an initial per-token KL around 0.04 for same-origin teachers and around 0.19 for the external-math-teacher setting. Their ratio is approximately 4.75, summarized as about fivefold.

Under the external teacher's policy-gradient run, reported entropy contracts from roughly 0.30 to 0.21 while math accuracy degrades. The top-$k$ variant is described as becoming severely unstable around step 18, with oscillations in KL and entropy.

These observations are consistent with a distributional mismatch. They do not isolate every possible cause. Changing teacher size also changes model behavior, potentially response style and target sharpness. A controlled recipe replacement is informative, but a causal decomposition would need further interventions.

Why punitive signals can dominate

At a sampled token, the advantage is $\log q-\log p$. If student-preferred continuations have much lower probability under the teacher, many advantages are negative.

The optimizer suppresses those continuations. If useful alternatives are rarely sampled, a sampled-token estimator may receive little positive evidence for them. In a limited or poorly tuned student, suppression can produce a narrow policy rather than a clean move to the teacher's successful modes.

Reverse KL contains an entropy term that resists simple collapse in the ideal fixed-context objective. Practical optimization still involves clipping, changing states, finite samples, limited capacity, and parameter coupling. Those conditions make observed entropy contraction compatible with the theory.

“Mode seeking” should therefore explain a possible tendency, not serve as a complete causal proof of the failure.

Why more target information can still destabilize

Top-$k$ gives multiple teacher alternatives at each prefix. One might expect that to solve the exploration issue. The reported failure shows that information alone does not guarantee fit.

A large mismatch can produce strong coordinated gradients on many selected alternatives. The student may not be able to represent the target behavior smoothly from its current parameters. Normalization changes probabilities outside the selected set, and later prefixes change as the student policy changes.

The paper's top-$k$ result therefore motivates examining optimization scale, teacher support mass, and target compatibility together. It does not prove that sparse support or the correction term caused the failure.

A compatibility diagnostic before a long run

Sample trajectories from the proposed student and score them with both the current and candidate teachers. Compare conditional KL estimates, signed advantage histograms, teacher entropy, captured top-$k$ mass, and outcome-stratified scores.

Look especially at positions around meaningful decisions: selecting a tool, committing to a proof strategy, adding a boundary check, or ending a response. An average over many ordinary syntax tokens can hide severe mismatch at these points.

Then run a small integration pilot with per-domain evaluation and monitor entropy and lengths. A good pilot cannot prove stability at scale, but it can catch obvious incompatibilities before an expensive run.

These diagnostics are recommendations. The paper does not report a universal numerical threshold below which a teacher is safe.

Possible remedies remain hypotheses

A lower learning rate, a gradual teacher transition, divergence-aware weighting, temperature changes, broader support, or preliminary alignment might improve transfer from an external teacher. Each changes the training recipe.

A useful study would vary one remedy at a time while holding compute and evaluation consistent. If a remedy restores performance, it would support a more specific explanation of the original failure than “different origins are always bad.”

The right takeaway is that compatibility belongs in teacher selection alongside competence. Shared origin is one effective tested route to compatibility in this paper.

Source notes

The replacement experiment, table, and approximate curve values are reported in MT-MOPD, Section 4.4.2 and Table 4. Interpretations of punitive signals and suggested remedies are analysis and proposals. No stronger universal claim is attributed to the experiment.

Previous: 10. Routing, Teacher Disagreement, and Shared Capacity · Contents · Next: 12. Teacher Services, Throughput, and the Cost of Supervision

12. Teacher Services, Throughput, and the Cost of Supervision

A teacher must process every student trajectory it supervises. Multi-Teacher MOPD makes this operation look like a reward service: the student sampler generates responses, and a domain-specific teacher service scores completed trajectories asynchronously.

This design can hide teacher latency behind ongoing sampling. It does not make teacher computation disappear. A useful cost analysis separates wall-clock time, accelerator work, memory, network traffic, and end-to-end pipeline cost.

Generation and prefill have different shapes

Autoregressive generation depends on earlier sampled tokens. Even with batching and optimized kernels, a long response has many sequential decisions. Teacher scoring receives the complete student trajectory, so causal next-token distributions can be computed across many positions in a prefill pass.

Both operations run a model. Prefill is often easier to batch and parallelize across sequence positions, while generation repeatedly extends a state. The exact balance depends on sequence length, hardware, architecture, kernels, and batch shape.

For agents, generation also pauses for environment operations. A compiler, test suite, or external tool may dominate the elapsed time. A teacher service can use those pauses to process other completed histories.

The reported asynchronous design

The paper deploys each specialist as a standalone prefill service outside the RL trainer. When a student's rollout finishes, its scoring request is dispatched to the corresponding service. Other student rollouts continue while the teacher computes the target.

The authors report nearly no measurable wall-clock overhead from teachers in their deployments. That is an empirical systems observation about sufficient service capacity and overlap. It is not a universal property of distillation.

A trainer still needs the target before applying the associated update. Poorly provisioned teachers can accumulate queues, stall batches, or increase rollout staleness.

A queueing condition

Let domain $d$ produce eligible student tokens at rate $\lambda_d$ and let its teacher service process them at sustainable rate $\mu_d$. A basic requirement for a stable scoring queue is

The rates must use comparable units and reflect realistic batching. Requests also incur per-request overhead, and long histories can create uneven service times. Token throughput alone may not capture latency tails.

If the sampler produces 20,000 math tokens per second while its math teacher can process 12,000, the queue grows even if the other teacher services are idle. Domain routing creates separate capacity demands.

Monitoring should include request age, queue depth, per-domain token throughput, timeout rates, and policy-version lag. Average scoring latency can look acceptable while a few long software histories block an update.

An ideal overlap model

Let $T_s$ be sampling time, $T_q$ teacher-scoring time, and $T_u$ update time for a batch. A fully serial loop costs roughly

With effective overlap and separate resources, an idealized cost is

plus dispatch, coordination, and tail effects. If $T_q<T_s$, teacher latency may be largely hidden. If the scorer shares the sampler's accelerators, contention can break this simple model.

The formula is a teaching model, not a measured scaling law for the paper. Real loops may overlap updates too, which introduces further memory and staleness constraints.

Full-vocabulary payloads can be enormous

Assume a hypothetical vocabulary size $V=150{,}000$, an eligible trajectory length $T=8{,}192$, and four bytes per probability. A full teacher distribution costs

for one trajectory before metadata. These illustrative values are not a claim about the paper's exact vocabulary or network configuration.

For sampled-token scoring, one four-byte log-probability per position costs 32,768 bytes, approximately 32.8 KB. For top-64 with four-byte IDs and four-byte values, the payload is

This explains why sparse targets can fit an external-service architecture. It does not mean sparse scoring is computationally as cheap as returning the payload: the model forward and output head still run.

Memory beyond transmitted targets

The teacher holds weights and attention activations or caches for scoring. The student optimizer holds parameters, gradients, optimizer state, and training activations. Rollout workers may maintain another copy of actor weights. Agent histories and queued records consume memory too.

Top-$k$ reduces target storage but does not proportionally reduce all these allocations. A small response payload can coexist with substantial scoring GPU memory.

Packing sequences improves utilization but must preserve causal boundaries and token alignment. If a packed request lets one trajectory attend to another, the teacher targets no longer correspond to the stated conditional distributions.

Total pipeline cost includes teacher development

Integration sample-efficiency curves do not include the specialist RL work that created the teachers. A fair end-to-end comparison should count

along with environment execution and evaluation as appropriate.

Parallel teacher training can reduce calendar time toward the duration of the slowest specialist run plus integration. It does not reduce the sum of accelerator-hours merely by running concurrently.

If specialists already exist for another purpose, integration's marginal cost may be the relevant quantity. If they are built solely to create one final student, the full pipeline cost matters. State the decision context before making an efficiency claim.

Peer conditioning adds a different cost

MR-MOPD lengthens the teacher's input with successful and failed peers. That changes prefill work and context memory even if the returned target payload remains sparse.

Its measured step times increase from 550 to 600 seconds on LCB and from 90 to 120 seconds on QA-Chemistry. The larger percentage on chemistry reflects a different bottleneck profile: coding reward execution is much more expensive in the reported timing table.

These numbers do not conflict with MT-MOPD's low-overhead service observation. The methods use different target construction, hardware organization, workloads, and timing comparisons.

A service contract that preserves meaning

A scoring request should identify the teacher checkpoint, tokenizer, chat template, sequence tokens, attention boundaries, sampled IDs or top-$k$ request, and target-distribution settings. A response should identify the request, version, positions, and normalized log-probabilities.

Retries must not accidentally score with a changed teacher version. A timeout should not silently substitute another specialist. Cache keys must include enough context to avoid reusing targets for a different prefix or privileged peer set.

These recommendations make numerical meaning auditable. A fast service returning the wrong conditional distribution is not a valid distillation component.

Source notes

The standalone asynchronous services and low reported overhead are from MT-MOPD, Section 3.3. MR timing values are from MR-MOPD, training-statistics appendix. Queueing, memory, and byte-count examples are engineering derivations with stated hypothetical assumptions.

Previous: 11. Why a Stronger Teacher Can Teach Worse · Contents · Next: 13. Reading the Multi-Teacher Experiments Correctly

13. Reading the Multi-Teacher Experiments Correctly

The main controlled study asks whether one student can recover the improvements of three specialist teachers. It compares multi-teacher distillation with mixed RL, sequential RL, offline finetuning, and parameter merging from a common SFT starting point.

The headline result is a normalized score of 0.9373. Understanding that number requires examining the raw benchmarks and the denominator used to define capability recovery.

What was evaluated

The starting model is Qwen3-30B-A3B-Base, followed by a broad custom SFT pass. All compared integration methods use that starting checkpoint according to the paper.

Math is evaluated on AIME25 and AIME26. Instruction following is evaluated on IFBench and IFEval. Software engineering is evaluated on SWE-bench Verified.

The “RL Teacher” row combines different specialist models: the math teacher supplies the math results, the IF teacher supplies IF results, and the SWE teacher supplies SWE results. It is not one teacher checkpoint that achieves all five entries simultaneously.

The full main table

The benchmark columns are percentages from MT Table 2. MT-MOPD has the strongest normalized aggregate among these integration runs, but it does not win every individual column. Mix-RL is better on AIME25. Off-Policy Finetune is better on IFBench. Task arithmetic is better on IFEval.

That pattern supports a capability-balance argument rather than a claim of universal per-benchmark dominance.

Define normalized headroom

For a domain $d$, let $s_d^s$ be SFT-student accuracy, $s_d^t$ specialist-teacher accuracy, and $s_d$ the integrated model's accuracy. The normalized score is

The SFT student has score zero; the teacher has score one. Values above one exceed the teacher's measured improvement; values below zero regress from the SFT student.

The headline averages the three domain scores uniformly. It is neither a raw accuracy nor the fraction of all possible abilities the model retains.

The reported table values can be reconstructed by averaging the two math benchmarks into a math accuracy, averaging the two IF benchmarks into an IF accuracy, normalizing each of those domain means, and averaging them with the normalized SWE score. This arithmetic matches the headline to reported precision.

Work through MT-MOPD

The math means are

Thus

For instruction following,

so

For SWE,

Their mean is approximately 0.9373. “93.7% recovery” therefore means recovery of measured teacher-over-SFT headroom under this aggregation.

What the 5.5-point lead means

The difference from Mix-RL is

That is 5.55 points on a normalized 0–100 headroom scale. It is not a 5.55-percentage-point increase in every benchmark or in an ordinary raw accuracy average.

The per-domain scores also show balance. MT-MOPD's spread is about 0.044, from approximately 0.910 to 0.954. A balanced normalized score can still hide a meaningful raw deficit on a particular benchmark, such as its 3.33-point AIME25 gap below the teacher.

Why normalization is useful and fragile

Raw domain headrooms differ. IF's mean headroom is 23.52 points while math's is 9.27. Equal raw averaging can reward a method more for the wider-headroom domain. Normalization asks how much of each specialist's improvement was recovered.

The denominator also creates sensitivity. If teacher and student differ by only one point, a half-point measurement fluctuation changes the normalized score by 0.5. If headroom is zero or negative, the ordinary interpretation fails.

Using different teachers changes the anchors. Normalized scores from two papers or two teacher bundles need not be comparable. Preserve raw scores and anchor definitions alongside the headline.

Evaluation sampling matters

For AIME25 and AIME26, the paper samples 32 responses per question and reports average accuracy, called avg@32. This estimates per-sample correctness across repeated attempts. It is not the probability that at least one of 32 responses is correct.

IFBench, IFEval, and SWE-bench Verified are evaluated once per task. The stated decoding temperature is 1.0 with no top-$p$ or top-$k$ truncation.

Repeated samples reduce generation noise on a fixed question set but do not turn a small benchmark into thousands of independent task types. Responses from one question share its difficulty. A confidence analysis should account for that clustering.

Sample efficiency and compute fairness

The paper describes IF approaching the teacher plateau after roughly 25K integration samples and SWE after roughly 30K, versus a much larger per-domain sample budget for Mix-RL. This is evidence of faster integration-stage progress in the plotted setup.

It does not include teacher-development cost in that comparison. Nor does equal sample count ensure equal token count or GPU-hours. MT integration and RL also differ in batch sizes, rollout group size, and Dynamic Sampling.

Those differences belong to the compared recipes. They help explain practical performance but limit causal claims that dense supervision alone accounts for the entire gain.

What the study supports

The controlled results support effective and relatively uniform integration of the tested specialists. They show weak simple weight averaging, stronger task arithmetic, and remaining domain-specific gaps in other baselines.

They do not settle all merge methods, all domain orders, all multi-task RL recipes, or all student sizes. Confidence intervals and repeated-seed evidence are not supplied for every MT comparison. Tiny differences should not be treated as established statistical superiority.

Source notes

All table values, the metric, and the sample-efficiency observations are from MT-MOPD, Sections 4.1–4.2, Table 2, and Appendix B. The explicit domain-mean reconstruction and fairness qualifications are the book's arithmetic and analysis.

Previous: 12. Teacher Services, Throughput, and the Cost of Supervision · Contents · Next: 14. Frontier-Scale Integration and Repeated Rounds

14. Frontier-Scale Integration and Repeated Rounds

A controlled experiment tests a method under relatively accessible conditions. An industrial application asks whether the same organizing idea is useful at much larger scale. A repeated-round experiment asks whether a unified student can become a better starting point for new specialists.

Multi-Teacher MOPD presents both kinds of evidence. Neither should be read as a complete scaling law or a proof that iterative improvement continues indefinitely.

The MiMo-V2-Flash application

The paper applies its three-stage pipeline to MiMo-V2-Flash, described in the section as a 309B-parameter model. Its teacher set covers math, code, instruction following, software engineering, and tool use.

The table compares an initial student, corresponding domain RL teachers, and the integrated model:

These are the reported Table 3 percentages. The teacher row again represents the relevant specialist for each benchmark, not a single unified teacher model.

Five entries exceed the teacher numerically. IFBench and SWE Verified remain below it. Several margins are small, and the table alone does not establish their statistical significance.

What industrial use adds

This evidence demonstrates that the method was used beyond the controlled Qwen3 experiment and that its reported outcomes are close to the specialist levels across several domains.

It does not expose all proprietary training data, compute budgets, serving configurations, or ablations. The table is not an independent reproduction. It also does not compare the industrial run with every controlled-study baseline under identical frontier-scale budgets.

A reader can conclude practical applicability in the reported production context. A reader cannot derive a precise cost-per-point scaling law or guarantee the same outcome for another 309B model.

How a student can exceed its teacher's score

A benchmark score is an empirical outcome, not a ceiling imposed by distillation. The student begins with its own prior knowledge, receives mixed-domain updates, and is optimized over many sampled trajectories.

If it is not an exact copy of a teacher, it can occasionally perform better on a finite benchmark. Cross-domain transfer, regularization, different sampling behavior, and measurement variability are possible contributors.

Exact matching to one teacher on all possible contexts under identical decoding would reproduce that teacher's distribution, rather than systematically exceed it. The actual pipeline does not establish such global equality. A benchmark gain therefore does not contradict the use of a teacher objective.

The evidence does not identify which mechanism explains each gain. A small positive delta alone is not proof of emergent cross-domain synergy.

The repeated-round idea

After one integration, use the unified model as the starting point for another set of domain RL teachers. Those teachers may improve further because they inherit a better general foundation. Then distill them into a student initialized from the same new starting point.

A schematic iteration is

This is a description of a workflow, not an update rule with a convergence guarantee. Each RL development phase introduces real environment feedback; the process is not merely copying a frozen teacher repeatedly.

The reported second round

The Qwen3 follow-up trains and distills new math and IF teachers only. It does not train a new SWE teacher or distill SWE in that round.

The teacher row's SWE entry is a carry-forward reference from the first student, not evidence of a newly trained second-round SWE specialist. The normalized scores retain the study's original reference scale.

The aggregate rises by approximately 0.049. AIME25, IFBench, and IFEval improve. AIME26 declines by 0.41 points and SWE declines by 0.20. Thus the second round improves the aggregate without improving every entry.

Retention and omitted domains

The small SWE change is encouraging because SWE receives no direct second-round RL or distillation. It suggests limited regression on this benchmark in this follow-up.

It does not establish general retention without rehearsal. Another domain mixture, a longer run, or a different benchmark could expose larger forgetting. The table lacks the broad evaluation needed for a universal retention claim.

If an engineering team updates only two domains, it should still evaluate every deployed capability. Indirect effects are a feature of shared parameters, whether positive or negative.

Why repeated rounds may eventually stall

A specialist can only improve if its training environment provides useful feedback and its model has remaining capacity. Integration can also lose some specialist gain. Eventually new RL headroom may shrink or integration errors may dominate.

No theorem in the paper guarantees monotonic improvement with unlimited rounds. The demonstrated follow-up is one round. It is useful evidence for extensibility, not a proof of indefinite self-improvement.

Track incremental teacher gain, integration recovery, retained-domain regressions, and total cost each round. A higher normalized score against fixed old anchors can obscure diminishing returns if the new teachers become only marginally better.

Source notes

The MiMo table and model-scale description are from MT-MOPD, Section 4.3 and Table 3. The follow-up is from Section 4.4.3 and Table 5. Explanations of teacher surpassing, iteration limits, and retention evaluation are interpretations and recommendations.

Previous: 13. Reading the Multi-Teacher Experiments Correctly · Contents · Next: 15. Multi-Rollout MOPD: Learning from Local Trial and Error

15. Multi-Rollout MOPD: Learning from Local Trial and Error

Multi-Rollout MOPD asks a different question from Multi-Teacher MOPD. Instead of selecting among specialist teachers, it improves the information available to a teacher scoring one student attempt. Other attempts on the same prompt become labeled evidence.

This chapter introduces that method independently. Its essential prerequisites are a student capable of generating several attempts, an evaluator that distinguishes successes from failures, and a teacher that can condition on those peer attempts.

From isolated attempts to a local group

For a prompt $x$, sample

Evaluate each trajectory with a verifier $R(x,y^{(i)})$. Given threshold $\tau$, partition the group:

Binary rewards make this straightforward. With graded rewards, the threshold encodes a substantive decision about what counts as a useful positive demonstration.

The teacher's task is still to supervise a target rollout $y^{(i)}$. The peers provide extra context, not a replacement trajectory to copy token by token.

Excluding the target from its peer set

For target $i$, define

This leave-one-out construction avoids presenting the full target as its own labeled demonstration. It matters because teacher forcing should predict target token $t$ from its prefix, not from a privileged copy of its complete future.

The target's identity and history may still inform context construction in a broad implementation. The method's mathematical notation writes a general context builder. A faithful operational version should specify exactly which target information the teacher can see and prevent unintended future-token leakage.

This is especially important when a context builder concatenates entire responses and then scores another response. Causal masking over the concatenated text does not prevent leakage if the target's full answer was already inserted earlier in the teacher prompt.

Peer-conditioned teacher distributions

Construct a context

where $v$ identifies the peer-selection strategy. The teacher target becomes

The student remains conditioned on $x,y^{(i)}_{<t}$. It learns from a teacher with more information than it receives directly.

A successful peer can show a valid reasoning relation. A failure can show a locally plausible mistake. Together they can help the teacher distinguish alternatives that look similar in ordinary language-model likelihood.

The loss over the rollout group

For each target,

The displayed algorithm averages over targets in the group. The method reports reverse KL for math and JS for other tasks.

Unlike MT's displayed per-response average, this MR expression is a token sum. An implementation may add normalization through its training framework, but the formulas should not be silently treated as identical. Loss aggregation affects the influence of response length.

The algorithm includes an $\alpha$ parameter, and implementation details report a mixing coefficient of 0.5. The rendered method does not provide a fully explicit final hybrid-objective equation tying every reported setting together. The reproduction discussion therefore treats the precise role of $\alpha$ as a detail to verify in code or configuration.

Positive and contrastive variants

The positive-only strategy supplies a primary successful peer and, when available, another successful peer. If only one successful peer is available, the method describes reducing to that primary solution.

The contrastive strategy adds a failed peer. The typical reported context contains two successful rollouts and one failed rollout. The following chapter examines what happens when these peers do not exist.

A failure must be labeled as failure. Otherwise its text may look like another demonstration to imitate. The teacher needs instructions that distinguish successful evidence from a plausible path to avoid.

This is contextual contrast, not a separately derived contrastive representation-learning objective. No negative-log-likelihood update on the failed peer is implied merely by placing it in the teacher context.

How self-distillation can improve a model

In self-distillation, teacher and student can use closely related model parameters. The teacher has privileged verified examples while the student does not. Their conditional distributions can differ despite their related weights.

A model asked “solve this task” and the same model asked “given these labeled solutions and mistakes, assess this continuation” are performing different conditional inference problems. The latter can be easier.

The new information ultimately comes from repeated sampling and the verifier. Self-distillation does not create correctness evidence from an unobserved source. If every attempt shares the same wrong assumption and the verifier accepts it, peer context can reinforce that error.

The reported implementation setting

The paper uses the verl framework, asynchronous vLLM rollouts, and FSDP-based optimization. It reports 32 prompts per batch, eight rollouts per prompt, actor learning rate $10^{-5}$, ten warmup steps, top-$k$ distillation setting 100, and bfloat16 weights.

Maximum prompt length is reported as 2,048 tokens; responses use 4,096 tokens, with 8,192 mentioned for math. The text also discusses extra context headroom and an 8,192-token total budget. These statements do not fully resolve how all longer math and privileged-context cases are packed or truncated. A reproduction should inspect the actual task configurations.

The top-$k$ value 100 belongs to the MR implementation. It should not be confused with MT's corrected top-64 objective; the papers do not establish identical sparse losses.

What local trial and error can teach

A failure may be informative because it is close to a success. A programmer can preserve an algorithm's overall shape but omit one check. Showing both examples may focus the teacher on that distinction.

Whether the teacher actually localizes the cause is an empirical matter. A failed response can contain many correct tokens and one decisive error. Its label does not imply that every token is bad.

The teacher converts coarse labels plus context into a dense conditional signal. This conversion is the core hypothesis tested by MR-MOPD.

Source notes

The groups, leave-one-out sets, teacher context, variants, and objective are from MR-MOPD, Sections 2–3. Implementation settings are from its experimental setup. The leakage analysis and distinction between contextual contrast and a contrastive loss are explanatory analysis.

Previous: 14. Frontier-Scale Integration and Repeated Rounds · Contents · Next: 16. Peer Selection, Availability, and Context Budgets

16. Peer Selection, Availability, and Context Budgets

Peer-conditioned distillation can only use evidence that the rollout group produces and the teacher can fit into its context. A recipe described as “two successes plus one failure” therefore has hidden availability conditions.

This chapter makes those conditions explicit. It also explains why increasing the number of peers is not automatically better and why peer selection is part of the learning algorithm.

The target changes how many peers are available

Suppose a group contains eight rollouts: one success and seven failures. For a failed target, the successful rollout is available as a positive peer. For the successful target, leave-one-out exclusion removes the only success, so it has no successful peer.

With two successes and six failures, a failed target can receive two positive peers, but a successful target has only one other success. The same group therefore supplies different evidence to different targets.

A report should distinguish group-level success counts from target-level context availability. The phrase “every group contains a success” does not imply “every target has a successful peer.”

An independent-sample model

Let $s$ be a fixed per-rollout probability of success for one prompt. Assume independent samples solely for this calculation. Among $N$ rollouts, the probability of at least one success is

For $N=8$ and $s=0.1$, this is approximately 0.5695. With $s=0.01$, it is about 0.0773. Even eight attempts rarely produce a positive peer on very difficult prompts.

For an individual target, its $N-1$ other rollouts supply at least one successful peer with probability

At $N=8,s=0.1$, this is approximately 0.5217. The probability of at least two successful peers is

approximately 0.1497 at the same settings.

These figures are illustrative. Real prompts have heterogeneous difficulty, and sampled responses can share failure patterns. A single global $s$ is usually a poor empirical model.

Requiring both two successes and a failure

A target has seven peers when $N=8$. Let $K$ be their number of successes. The compact contrastive recipe requires $2\le K\le6$: at least two successes, with at least one failure remaining.

Under the independent model,

At $s=0.1$, this is almost the same 0.1497 because all seven peers succeeding is extremely unlikely. At $s=0.9$, the requirement is satisfied with probability about 0.5217: failures have become the scarce evidence.

There is a curriculum implication. Hard prompts lack positives; easy prompts lack negatives. A fixed context recipe interacts with the student's changing success distribution over training.

All-failure groups are a central case

The MR signal analysis reports that 57% of prompts in the analyzed base-model LCB rollout set have no successful rollout. That observation makes fallback behavior essential, not an obscure edge case.

Possible fallbacks include using a failure-only context, scoring without peers, obtaining external positive evidence, resampling, or skipping the group. These choices create different data distributions and compute costs.

The method text clearly describes reducing positive context when only one positive peer is available. It does not fully specify every all-failure, all-success, and insufficient-peer case for all experiments. The book therefore does not invent one universal fallback and attribute it to the paper.

An implementation should record the actual context mode for each target. Without that information, a method named after a mixed context may spend much of its training in another mode.

What makes a failure informative?

A failed attempt is not uniformly negative evidence. A syntax error, a timeout, an incorrect algorithm, and a formatting omission are different failure types. A teacher that sees only “failed” must infer the cause from text and any available feedback.

A near-success failure can isolate a missing constraint. A radically unrelated response may be less useful. A very long failure can consume context without adding much information.

Selecting failures by similarity to a success, error category, or verifier trace is a plausible improvement. Such selection can also introduce bias: it may overemphasize familiar error families and miss new ones. These ideas are proposed extensions rather than demonstrated MR components.

Multiple successes can show alternatives

Two successes may differ in strategy, implementation, or explanation. They can prevent the teacher from treating one wording as the only acceptable solution.

But two nearly identical successes may supply little extra evidence. A selection rule that rewards diversity needs a definition of diversity. Lexical distance can confuse variable renaming with an algorithmic difference. Structural distance can overlook a decisive constant or comparison operator.

The relevant goal is not maximum difference in text. It is useful complementary evidence about valid reasoning and common mistakes.

Context is a finite budget

The teacher input contains the task, peer demonstrations, labels, and the target prefix. If each peer is long, raw concatenation can dominate the context window.

Truncating the end of a peer may remove its final answer or the line that caused failure. Truncating the target prefix changes the state being supervised. Any truncation policy therefore affects semantic validity, not merely memory use.

Peer summaries could reduce cost, but summaries can omit decisive details or introduce new mistakes. Summary-based variants appear in unused or revised source material; the active v2 method studies positive-only and success-failure contexts. This book does not present a summary variant as an established v2 result.

The ablation is not “more peers always wins”

The reported compact two-success-plus-one-failure setting performs strongly on both chemistry and LCB. All-eight-peer settings do not improve LCB over that compact setting, while a longer-run eight-peer chemistry setting has a higher score.

The eight-peer rows include different time budgets. They therefore do not isolate peer count independently of training duration. They are useful observations about a tradeoff, not a clean monotonic scaling experiment.

A defensible study would vary peer count while holding generated tokens, target count, context policy, and total accelerator time as comparable as possible. Report both quality and availability.

Source notes

Peer recipes and the one-positive reduction are from MR-MOPD, Section 3. The 57% observation and ablation are from its teacher-signal analysis. The binomial formulas, fallback taxonomy, and peer-selection proposals are independent analysis.

Previous: 15. Multi-Rollout MOPD: Learning from Local Trial and Error · Contents · Next: 17. Privileged Information and the Limits of Self-Teaching

17. Privileged Information and the Limits of Self-Teaching

A teacher can be stronger because it has better parameters, because it receives better information, or because both are true. Multi-Rollout MOPD emphasizes the second source. The teacher reads labeled peer attempts unavailable to the student when it first attempts the task.

This creates a form of privileged-information learning. Understanding the information asymmetry explains why self-distillation can help and why it still has limits.

The conditional problems differ

The student predicts

The peer-conditioned teacher predicts

Even if $\phi$ and $\theta$ are equal at some moment, these distributions need not match because the conditioning inputs differ. A verified example can resolve ambiguity that the original task alone leaves difficult to interpret.

For instance, a successful code peer may demonstrate that an identifier means a person index rather than a displayed label. A failed peer may show the exact confusion. The model has not gained new parameters before scoring; it has gained evidence.

The original SDPO method already uses successful rollouts as implicit feedback when only scalar rewards are available. MR's contribution is the particular structured peer-context treatment, especially explicit failed-peer evidence alongside successes; it should not be read as introducing successful-rollout supervision from scratch. SDPO primary paper.

Where the information originates

The student samples alternatives. The verifier labels them. The teacher interprets the alternatives and labels. Distillation transfers that interpretation into future student behavior.

The loop uses an external correctness signal even when the teacher is a self-teacher. “Self” refers to the model relationship, not to a closed system without feedback.

A useful accounting separates generated candidate information from verifier information. Candidate text alone may contain many plausible solutions. The success/failure labels tell the teacher which evidence to trust under the evaluator's criterion.

Why context distillation can generalize

Across many tasks, the student can learn recurring patterns: preserve all constraints, maintain identifier types, use the right API arguments, and avoid common mistaken branches. It need not see the same peer demonstrations at deployment if these patterns become encoded in its parameters.

Some privileged information may be task-specific and impossible to infer later. If the teacher is given a secret answer unrelated to the visible prompt, no student can recover it universally without that information. Distillation can average or approximate such targets, but cannot defeat missing information in principle.

Peer evidence is useful when it reveals learnable structure or helps produce better targets on the training distribution. Generalization beyond that distribution still requires evaluation.

Dense supervision does not become ground truth

The verifier gives a trajectory label. The teacher converts it into token-level predictions. Those predictions are inferred guidance, not directly observed causal responsibility for each token.

A failed program can contain correct syntax, a sound algorithm, and one wrong boundary condition. If the teacher suppresses every token merely because the program failed, it loses useful information. A good conditional teacher distinguishes the error from the surrounding valid structure.

Conversely, a successful program can exploit a weak test suite. A labeled success may then become misleading positive evidence. The teacher's extra context can amplify evaluator weaknesses rather than correct them.

Leakage has several meanings

During training, the teacher intentionally sees verified peers. That is authorized privileged supervision within the method. It is not automatically an evaluation leak.

Leakage becomes a concern if held-out answers enter training contexts, if the full target response is supplied while scoring its own earlier tokens, or if evaluation peers contain information the deployment task would not supply and the result is presented as ordinary single-attempt performance.

Keep teacher-time privilege separate from student-time evaluation. Evaluate the final student without hidden peers if that is the intended deployment setting. Record how task splits and any resampling are constructed.

Teacher surpassing needs matched conditions

MR-MOPD's teacher-student experiments compare a 4B student and 14B teacher. The peer-conditioned student can exceed the larger teacher's reported ordinary benchmark scores.

The supervising teacher has richer training-time context than the free-running teacher used for its own evaluation. The student also receives many optimization updates. Therefore this comparison does not show that a smaller model reproduces every ability of a larger one or that it exceeds a teacher under every information condition.

It shows that the trained student obtains stronger measured task outcomes in the reported setting. That is valuable without turning it into an unrestricted intelligence claim.

Is the teacher frozen, delayed, or identical to the actor?

“Self-distillation” can mean a current actor with privileged context, a delayed copy, an exponential-moving-average teacher, or another related checkpoint. These designs differ in target stability and information flow.

The MR paper's conceptual formulation permits a teacher $q_\phi$ and reports a self-distillation configuration. It does not fully resolve every parameter-synchronization and mixing detail in the manuscript alone. An exact reproduction should inspect the linked implementation rather than assume one universal self-teacher update scheme.

This uncertainty does not prevent understanding the peer-conditioning mechanism. It limits claims about exact operational equivalence to a specific codebase.

No-success prompts and exploration

When no peer succeeds, the teacher lacks verified positive evidence from that group. It may still use pretrained knowledge, failure context, or environment feedback, but the intended positive-negative contrast is incomplete.

Generating more attempts increases the chance of finding positives at a cost. Resampling only until success biases the training distribution toward solvable prompts and consumes variable compute. External reference solutions add a different information source.

A good experiment reports how often each situation occurs. Otherwise improvements can be hard to attribute to the advertised contrastive context rather than fallback behavior or extra search.

The broader lesson

Distillation quality depends on the teacher's information set as well as its weights. A compatible model with precise local evidence can be a better supervisor than a more capable model with an unhelpful context.

This lesson complements MT-MOPD's compatibility result. One paper improves teacher selection; the other improves teacher conditioning. Neither makes correctness automatic, but both move attention from model size alone toward the actual supervision problem.

Source notes

Peer-conditioned distributions and self-distillation are from MR-MOPD, Sections 2–4. Teacher-student outcomes are discussed in its final experimental subsection. The information-theoretic boundaries, leakage taxonomy, and synchronization distinctions are analysis.

Previous: 16. Peer Selection, Availability, and Context Budgets · Contents · Next: 18. What the Multi-Rollout Results Show

18. What the Multi-Rollout Results Show

Multi-Rollout MOPD evaluates peer-conditioned supervision across code, math, scientific QA, and tool use. Its central comparison is whether labeled peers improve over scalar-reward RL and an existing self-distillation baseline under the reported training budget.

The results are encouraging but heterogeneous. Some large gains are relative to a weak self-distillation result, some are modest increments over a strong on-policy baseline, and not every benchmark column favors MOPD.

Mean@8 and pass@8 answer different questions

Mean@8 averages correctness across eight attempts per question and then across questions. It measures the quality of a randomly selected attempt under the evaluation sampling procedure.

Pass@8 measures whether at least one of eight attempts succeeds, or an estimator of that quantity when more samples are available. It measures the usefulness of multiple attempts together. A larger pass@8 can coexist with a low mean@8 if most attempts fail but one occasionally succeeds.

For $n$ sampled candidates with $c$ successes, the familiar estimator for $k\le n$ is

When $n=k=8$, this reduces to an indicator that at least one sampled candidate succeeded. An exact reproduction should verify the paper's evaluator aggregation rather than infer it solely from the label.

Neither metric can be directly compared with MT's avg@32 or an unrelated greedy accuracy without matching the protocol.

LiveCodeBench results

The study uses LCB v6, described as 131 contest problems from February through May 2025. The active v2 main table reports:

The gain over SDPO is 8.17 mean@8 points for 4B and 12.33 for 8B. Pass@8 improves by 2.25 and 2.89 points respectively. Mean and pass improve by different amounts, suggesting that per-attempt reliability and group success are not moving identically.

The archive contains an older commented-out table with different mean values. This book uses the active compiled table rather than the commented source. The source audit records that distinction so numbers remain reproducible.

Math does not show universal dominance

All methods in the math table start from Qwen3-4B and use the stated generation budget:

MR-MOPD strongly improves over SDPO, whose AIME performance falls below the base. But GRPO is slightly better on both AIME25 columns and on HMMT February pass@8.

The result supports an improvement over the tested self-distillation baseline and strong performance across several tasks. It does not justify “MOPD beats GRPO on every math benchmark.” Nor can one weak SDPO run establish that feedback-based self-distillation is intrinsically unsuitable for all math tasks.

The authors suggest sparse binary math feedback explains SDPO's weakness. That is a plausible interpretation, not a separately isolated causal result.

Science and tool use

The science table uses Qwen3-8B:

The main ToolUse table gives 59.11 for the base, 60.32 for GRPO, 63.45 for SDPO, and 66.73 for MR-MOPD. These are the reported main-table percentages.

An appendix three-run table reports a different SDPO chemistry mean, 73.22 ± 2.5, versus the main table's 62.91. This materially changes the apparent chemistry margin. The next chapter treats it as an unresolved source discrepancy rather than selecting whichever comparison looks stronger.

The peer-context ablation

Positive evidence matters. Failure-only performs poorly. Adding negative evidence to successes often improves mean performance, and the compact mixed context is strong on LCB.

The one-success-plus-one-failure LCB pass score is lower than the one-success score despite its higher mean. The longer eight-peer chemistry run is better than the compact mixed setting. Thus the data do not support a simple monotonic ranking for every metric.

The differing time budgets also limit peer-count causal claims. More context can increase per-step cost; equal elapsed time can imply different numbers of updates.

The larger-teacher experiment

A separate study uses Qwen3-4B as student and Qwen3-14B as teacher:

The peer increment over teacher-student OPD is 6.81 mean points on ToolUse and 1.00 on LCB. LCB pass changes by only 0.05, which should not be called a substantial established gain without uncertainty estimates.

The paper's “GKD” baseline is described operationally as teacher-generated offline SFT. That label should not be taken as a full characterization of every method in the broader GKD literature.

Budget and uncertainty

The manuscript states matched rollout budgets and optimization protocol across methods. That is useful, but peer conditioning still changes teacher context and measured elapsed time. Generation budget equality is not complete compute equality.

Science has three-run standard deviations in an appendix. Comparable uncertainty reporting is not supplied for every code, math, and teacher-student comparison. Small deltas should be treated as numerical observations.

The evidence is strongest when a large margin appears across multiple settings and metrics with a clear protocol. It is weaker when a claim depends on one tiny difference, one selected case, or inconsistent source values.

Source notes

Tables and task descriptions are from MR-MOPD, Sections 4–6 and the training-statistics appendix. Values are taken from active v2 TeX, particularly text/exp_1.tex, text/exp.tex, and text/exp_3.tex. Metric distinctions, delta calculations, and uncertainty qualifications are analysis.

Previous: 17. Privileged Information and the Limits of Self-Teaching · Contents · Next: 19. Auditing Teacher Signals, Diversity, and the Source

19. Auditing Teacher Signals, Diversity, and the Source

Better downstream accuracy is the main result. A mechanism analysis asks whether peer context improves the teacher signal that might produce that result. MR-MOPD studies teacher-score agreement with verifier outcomes and measures output diversity.

These analyses add evidence, but their definitions and aggregation rules matter. This chapter explains the metrics and records source issues that a reader should resolve before treating every plotted number as a clean statistical quantity.

The teacher score being evaluated

For a fixed candidate response $y^{(i)}$, the teacher score is average token log-probability:

The verifier reward $r_i$ remains fixed while the teacher context changes. This is a useful design: candidate texts and outcomes are held constant, so changes in score quality can be associated with context construction.

Average token likelihood is not a direct correctness probability. It also reflects fluency, style, length-related effects, and familiarity. The analysis tests how well that score orders candidates rather than assuming it is an oracle.

Ranking and discrimination metrics

Spearman correlation compares ranks of teacher scores and rewards. With binary rewards, many reward ties occur. Kendall-style agreement compares concordant and discordant pairs; the appendix explicitly ignores tied pairs in its formula.

Pairwise accuracy considers pairs with unequal reward and reports how often the teacher orders them correctly. Success AUC compares positive-negative pairs and gives half credit for tied teacher scores. Point-biserial correlation is Pearson correlation between continuous scores and binary success labels.

For binary rewards and no teacher-score ties, pairwise accuracy and AUC coincide within a valid prompt. The reported table's close or identical values reflect that relationship. They are related views of discrimination, not wholly independent confirmations.

The reported offline table

The mixed context leads the five displayed ranking/association columns. Failure-only gives the lowest Brier value. Those observations are faithful to the table; their interpretation needs the qualifications below.

Prompts with no class variation

For a prompt with no successes, AUC's denominator $|\mathcal P||\mathcal N|$ is zero. Pairwise accuracy also has no unequal-reward pairs. Spearman and point-biserial correlation are undefined when rewards are constant. All-success prompts have analogous problems.

The paper reports 57% all-failure prompts in the analyzed LCB set. Therefore its “average over prompts” depends materially on whether undefined values are excluded, replaced with zero, or handled another way.

The manuscript definitions do not fully specify that aggregation convention. This matters especially for the low average AUC values. An AUC of 0.4733 is below the ordinary chance reference of 0.5 for a valid mixed-label population, but an average that inserts zeros for undefined prompts cannot be read using that reference without adjustment.

The authors attribute low baseline metrics partly to all-failure prompts. That explanation itself signals the need to inspect the implementation. The book does not infer valid-prompt AUC or a hidden denominator from incomplete information.

A reproduction should report valid prompt counts, exclusion rules, and both prompt-weighted and pair-weighted metrics where appropriate.

Sigmoid Brier is not calibrated success probability

The appendix maps the average log score to $\sigma(s_i)$ and computes squared error against binary reward. Since normalized log-probabilities are nonpositive, their average is nonpositive, and

The mapping cannot express a success probability above one half unless another transformation is applied. It is a fixed pseudo-probability mapping, not a fitted calibration model.

On a mostly unsuccessful dataset, low predictions can produce a small Brier score even when ranking is poor. The failure-only condition's low Brier therefore does not prove it is a better teacher.

A stronger calibration study would fit a transformation on a separate split and evaluate reliability and proper scoring on held-out data. It would keep discrimination and calibration separate. That is a proposed improvement rather than a reported analysis.

Diversity measurements

The paper reports:

Distinct-$n$ is unique $n$-grams divided by total $n$-grams. The appendix defines it across concatenated response token streams, while the main discussion describes within-response lexical richness. The appendix's operational definition is the more precise reference for reproduction.

AST distance compares multisets of Python node types. It detects some structural variation but ignores many semantic distinctions. Two programs can use the same node types and different incorrect conditions. Variable renaming can change lexical diversity without changing the algorithm.

Failed parses are excluded from AST measurement. This creates selection effects unless parse rates are reported. The code extractor keeps the last fenced block and falls back to raw text; this can also influence results.

The measurements support increased variation under these representations. They do not prove increased diversity of correct algorithms or establish that failed-peer context alone caused the entire increase.

Case studies are illustrations, not aggregate causal proof

The appendix shows identifier-composition errors, a dropped middle-character constraint, and output-format failures. They make the proposed mechanism concrete.

Selected examples cannot quantify population-wide error reduction. Some comparisons also combine correctness with parser or format behavior. A broader error taxonomy, blinded labeling, and matched ablations would strengthen the causal claim.

One training-time case has one successful rollout and seven failures while discussing supervision of a successful target. Under the stated leave-one-out rule, that target lacks a successful peer. The manuscript does not fully reconcile every detail of this case with context construction. This warrants implementation inspection; it is not sufficient evidence to allege target leakage.

Source inconsistencies to preserve explicitly

These issues do not erase the reported downstream gains. They limit numerical and operational conclusions that depend on the unresolved details.

Source notes

Metric definitions, tables, diversity extraction, and case studies are from MR-MOPD's analysis sections and appendices, especially text/exp.tex, text/appendix.tex, and text/case_study.tex. The aggregation and sigmoid critiques are mathematical analysis. Source discrepancies are observations from the retrieved active v2 TeX.

Previous: 18. What the Multi-Rollout Results Show · Contents · Next: 20. Worked Examples from Task to Update

20. Worked Examples from Task to Update

This chapter connects the pieces through small examples. The tasks, distributions, and calculations are educational constructions unless explicitly identified as a paper result. They are deliberately small enough to inspect by hand.

The aim is to see what happens between a task outcome and a parameter update, and to distinguish operations that are often compressed into the phrase “teacher supervises the student.”

A task with two identifier spaces

Three people have displayed labels. Let $Q[i]$ be the displayed label worn by person $i$, and let $P[i]$ be the person that person $i$ looks at. For each displayed label, output the displayed label of the person its wearer looks at.

Use one-based person IDs and the following data:

The displayed-label-to-person inverse is: label 1 belongs to person 2, label 2 to person 3, and label 3 to person 1.

Then:

The answer is $[3,2,1]$. The composition is

The task is inspired by an error family in MR's case study, but these values and the explanatory table are constructed for this book.

Plausible successful and failed attempts

A successful attempt explicitly constructs $Q^{-1}$ and follows the three-step composition. Another success might use a dictionary instead of an array. A failed attempt might return the looked-at person's ID without mapping through $Q$, or confuse $P[i]$ with the wearer of $Q[i]$.

A verifier executing enough cases can label these attempts. A weak test that happens to use identity $Q$ would miss the distinction between person IDs and displayed labels. Verifier design therefore determines whether the peer evidence is informative.

For an eight-attempt group, imagine two successes and six failures. A failed target can receive both successes and one other failure. A successful target can receive only the other success, because the target itself is excluded.

This availability difference is part of the training record. A context builder should not duplicate the target to manufacture the advertised peer count.

The teacher sees contrast; the student sees the task

For a failed target, the teacher prompt can explain that two peer programs passed and one peer program failed, then present the target prefix for causal scoring. The teacher may infer that preserving identifier types is the decisive issue.

The student does not directly read those peers during its original attempt. Its distribution at the prefix reflects only the task and its own history. Distillation teaches it to prefer a better continuation in future attempts.

The teacher is not necessarily asked to emit a natural-language critique. It can supply next-token probabilities. A critique is an optional additional product, not required by the divergence objective.

A three-option local distribution

Compress a meaningful decision into three toy alternatives:

• A: keep the mistaken identifier composition.
• B: use the correct inverse-and-forward composition.
• C: edit an unrelated part of the program.

Suppose

These labels represent a toy vocabulary, not actual multi-token action probabilities in a real tokenizer. The simplification allows direct arithmetic.

The reverse KL is

The sampled-token advantages are

The mistaken branch receives negative pressure when sampled. The correct branch receives positive pressure. The unrelated branch has no immediate sampled-token advantage at this context.

The full local gradient

For full-vocabulary reverse KL with softmax logits $z$, the gradient is

Numerically, it is approximately

The components sum to zero up to rounding, as expected from softmax's invariance to a common logit shift. Gradient descent lowers A's logit and raises B's. C's logit also rises slightly because the probabilities are coupled, even though its own ratio is one.

The sampled-token estimator reproduces this gradient in expectation under the local assumptions. One particular sampled token produces a noisier update. A large batch averages such samples across positions and tasks.

The top-two corrected loss

The teacher's top two options are B and C. The corrected partial loss is

This number is lower than the full KL because the A coordinate is unscored. It is not a directly comparable estimate of the same complete discrepancy.

Matching B to 0.7 and C to 0.2 forces A to 0.1 in this three-option example because only one tail option remains. With many tail tokens, matching selected masses would leave their internal allocation unspecified.

The correction keeps the selected-coordinate objective well behaved. It does not reveal which unselected token should carry tail mass.

An MT-MOPD trace for the same task

Suppose the task is classified as coding or software engineering under a particular domain schema. The student generates its own attempt. The designated specialist reads that attempt and scores its prefixes. MT-MOPD uses those scores without requiring peer demonstrations.

A different task asking for a constrained short answer is routed to an IF teacher. A math problem is routed to a math teacher. The student's weights receive all those updates over the mixed dataset.

The routing label determines the supervisor. It does not necessarily appear in the final user's prompt or become an inference-time teacher call.

A hypothetical combined trace

One could first route this task to its domain specialist and also provide verified peers. The target would then be

This combines teacher selection with context enrichment. It is a proposed extension, not a joint method validated by the two supplied papers.

Its compute includes generating several attempts, verifying them, constructing target-specific peer contexts, and querying the specialist. Compatibility and peer quality can both matter. A specialist trained to generate solutions may not necessarily interpret failed demonstrations well.

A combined experiment would need ablations for specialist routing alone, peers alone, and both together. Simply adding their separate reported gains would be unjustified.

What the example teaches

The verifier labels complete attempts. The teacher translates evidence into local distributions. The loss converts distribution differences into gradients. The optimizer changes shared parameters. Fresh sampling then tests whether those changes produce better trajectories.

Each transition can fail independently. A wrong verifier label, leaking context, misnormalized teacher payload, wrong token alignment, or unstable optimizer can break the chain. That is why understanding an end-to-end trace is more useful than memorizing a single KL equation.

Source notes

The identifier-space error family is discussed in MR-MOPD's case-study appendix. All concrete arrays, toy distributions, gradients, and combined-method traces here are authored examples. MT routing and corrected top-$k$ follow MT-MOPD, Section 3.

Previous: 19. Auditing Teacher Signals, Diversity, and the Source · Contents · Next: 21. An Implementation Blueprint with Explicit Assumptions

21. An Implementation Blueprint with Explicit Assumptions

This chapter translates the research ideas into a practical design. It is an implementation blueprint, not a claim to reproduce either paper's undisclosed operational details. The distinction is important because several choices required by real software are not fully specified in the manuscripts.

The blueprint supports two modes: domain-routed Multi-Teacher distillation and peer-conditioned Multi-Rollout distillation. A proposed combined mode can reuse components, but needs its own evaluation.

Define the experiment before the services

Write down the student checkpoint, tokenizer, task distributions, teacher identities, sampling distribution, target divergence, loss weighting, and evaluation protocol. For MT, define domain routing and teacher development. For MR, define the verifier, threshold, peer-selection rule, and every insufficient-peer fallback.

Also specify whether the optimizer uses one update per rollout, how the teacher is frozen or synchronized, and whether the sampling policy can lag the actor. These choices determine the objective the system approximates.

Do not start with the name of a framework and assume it resolves these questions. Framework defaults can silently add normalization, clipping, KL penalties, or repeated updates that change the recipe.

Use an auditable record schema

A conceptual record might contain:

record_id, prompt_id, domain, split
student_checkpoint, behavior_policy_version
tokenizer_hash, chat_template_hash
token_ids, attention_boundaries, actor_loss_mask
behavior_logprobs, sampling_settings
termination_reason, generated_token_count
verifier_version, reward, verifier_feedback
teacher_checkpoint, teacher_request_id
peer_ids, peer_labels, peer_context_mode, truncation_metadata
teacher_sampled_logprobs or teacher_topk_ids_and_logprobs

This is a suggested schema. Not every field is needed for every mode. MT can omit peer fields; MR may omit domain routing. The point is to retain enough evidence to reconstruct the conditional target and distinguish intended changes from infrastructure drift.

Build the scorer around supplied trajectories

The teacher scorer should accept a sequence to evaluate, not only a request to generate a completion. It must return probabilities at positions aligned with the supplied student tokens.

For MT, render the task and student history using the compatible chat format. For MR, render labeled peers before the target continuation, while excluding the target's complete future. Track the offset between the augmented teacher context and the unaugmented student target.

Position $t$ in the student response will usually not have the same absolute index in the teacher input after peers are added. Alignment must be represented explicitly. Gathering teacher position $t$ by raw index would score the wrong token.

Verify probability normalization

For sampled-token PG, return normalized log-probabilities of the student-selected IDs. For corrected MT top-$k$, return selected teacher IDs and their full-vocabulary log-probabilities.

Do not softmax only over returned teacher logits unless the target intentionally uses a renormalized subdistribution. Do not compare raw logits from two models. A logit offset can change their difference without changing either probability distribution.

For JS, document the sparse approximation and any tail treatment. The MR paper's distillation_topk=100 setting alone does not fully define the distributional approximation.

An MT orchestration loop

for step in training_steps:
    prompts = draw_domain_mixture()
    trajectories = sample_student(prompts, current_policy_version)
    targets = await score_with_routed_frozen_teachers(trajectories)
    actor_outputs = recompute_student_scores(trajectories)
    loss = chosen_mt_loss(actor_outputs, targets, trajectories)
    update_actor(loss)
    synchronize_sampler_with_documented_lag()

The loop is conceptual. Real systems overlap requests, pack trajectories, and shard models. Those optimizations should preserve teacher identity, causal boundaries, and the defined rollout-age policy.

If a scoring service is unavailable, retry or mark the record as failed. Quietly routing its task to a different teacher changes the learning target.

An MR orchestration loop

for step in training_steps:
    prompts = draw_prompts()
    groups = sample_student_groups(prompts, n_rollouts=8)
    labels = verify_groups(groups)
    target_records = []
    for group in groups:
        for target in group:
            peers = exclude_target(group, target)
            context = build_labeled_context(peers, explicit_fallback_policy)
            target_records.append(align_student_and_teacher_views(target, context))
    targets = score_peer_conditioned_teacher(target_records)
    loss = chosen_mr_loss(target_records, targets)
    update_actor(loss)

Store the chosen peers and the actual fallback mode. If the context budget truncates a peer, retain what was removed. Otherwise the experiment cannot distinguish intended contrast from incomplete demonstrations.

Test semantics before scale

Use a tiny synthetic vocabulary to compare analytic and automatic gradients. At $p=q$, full KL and corrected sparse loss should be zero within numerical tolerance. For a selected coordinate, the corrected loss should be minimized at equality, while the naive partial expression need not be.

Compare teacher-service scores to a trusted local forward pass on a few fixed sequences. Test one-token shifts, packed sequence boundaries, EOS handling, role masks, and peer offsets. For an agent record, confirm that tool observations have zero actor-loss weight under the chosen recipe.

These are meaningful correctness checks because a mistake can corrupt an expensive training run. They test mathematical and alignment properties rather than merely mirroring a helper function's implementation.

Watch policy age and asynchronous backpressure

A rollout generated many updates ago is not a fresh current-policy sample. Limit acceptable lag or apply a documented correction and stabilization scheme. Log lag distributions rather than a single average.

Apply backpressure if teacher queues grow. Unbounded collection can consume memory and increase staleness. Batching should improve utilization without making target delivery so delayed that the intended on-policy regime is lost.

A checkpoint synchronization event should update metadata atomically. If weights and version labels become inconsistent, behavior-probability comparisons cannot be trusted.

Treat verifiers as versioned software

A reward function can change through parser fixes, test updates, tool-image changes, or timeout settings. Store its version. A new parser can turn previous format failures into successes without any model improvement.

For MR, label changes also alter peer contexts. A verifier bug affects both outcome learning and teacher supervision. Run a small manually reviewed sample to establish that positive and negative examples mean what the teacher prompt says they mean.

Deliver the right artifacts

A useful experimental release includes configurations, checkpoint identities, source revisions, sampling settings, evaluator versions, aggregation rules, training curves, and representative aligned records. Private training data may prevent full reproduction, but the remaining operational specification can still be precise.

This book supplies a source-based explanation and a blueprint. It does not claim that pseudocode is an executed implementation or that the published scores have been independently reproduced.

Source notes

MT's routing and external scoring service motivate the first loop; MR's group partition and peer context motivate the second. See MT-MOPD, Section 3 and MR-MOPD, Sections 3–4. Schemas, tests, and failure-handling rules are proposed engineering design.

Previous: 20. Worked Examples from Task to Update · Contents · Next: 22. Designing a Reproduction That Answers the Right Question

22. Designing a Reproduction That Answers the Right Question

A reproduction should do more than obtain a score near a published table. It should specify which claim it tests, match the relevant information and compute conditions, and expose deviations from the original recipe.

This chapter organizes the known settings and unresolved details for both papers, then proposes an evaluation program. It assumes the reader understands student rollouts and teacher scoring, but does not assume access to proprietary data or frontier-scale hardware.

Choose the claim and scale

For MT, a modest study can test whether a shared-origin teacher set integrates more uniformly than offline imitation or mixed RL. It need not claim to reproduce MiMo-V2-Flash.

For MR, a study can test whether labeled failed peers add value beyond successful peers at a fixed rollout budget. It need not initially cover every task family.

Use a model and environment small enough to debug, but retain the structure relevant to the claim. A toy arithmetic benchmark cannot establish software-agent transfer, while a tiny vocabulary can still validate loss gradients.

Reported MT settings

Do not infer an integration learning rate from the specialist RL values. Dataset names do not provide exact selected prompts, filtering, split rules, or distilled targets.

Reported MR settings

The manuscript does not completely define teacher synchronization, every fallback, the hybrid coefficient's operational role, sparse JS handling, or all packing/truncation cases. The linked repository is a source to inspect for these details, not evidence already reviewed by this book.

Use several fairness axes

Equal prompts, equal generated trajectories, equal generated tokens, equal gradient updates, equal accelerator-hours, and equal elapsed time are different controls. No single one answers every efficiency question.

For MT, report teacher-development and integration costs separately and together. For MR, report verifier cost and longer teacher-context cost. If an experiment uses more compute to find extra successful peers, count that search.

Match starting checkpoints and held-out evaluation. Explain every recipe deviation. A result obtained with a different student, new prompt filtering, and a changed evaluator is an adaptation rather than a literal reproduction.

Separate ablations from whole-recipe comparisons

A whole-recipe comparison asks which pipeline works better in practice. An ablation asks which component causes a difference. Both are useful but require different controls.

For MT, useful ablations include same-origin versus external teachers, PG versus corrected top-$k$, several $k$ values, mixtures, and teacher-choice quality. Varying only a teacher while leaving hyperparameters fixed tests replacement robustness; separately retuning each teacher tests best achievable transfer.

For MR, compare no peers, one success, two successes, success plus failure, two successes plus failure, and carefully budgeted larger contexts. Report fallback frequency for each setting. A context-count comparison without available-peer statistics is incomplete.

For a proposed combined method, use a factorial study with routing present/absent and peers present/absent. Separate gains cannot simply be added.

Evaluate conjunctions and retention

Test domain-specific benchmarks and requests requiring several skills at once. A model that passes math and formatting tasks separately may fail when a mathematical answer must obey a strict schema.

Measure general SFT abilities that are outside the integration mixture. In repeated rounds, evaluate omitted domains even when they receive no direct training. Retention is a result to measure rather than an assumption.

For tools, score valid syntax, correct selected tool, correct arguments, successful environment outcome, and side effects separately where the task permits. A structured call can be syntactically valid and semantically wrong.

Quantify uncertainty at the task level

Report multiple training seeds when feasible. For repeated generations, resample or bootstrap at the prompt level rather than treating every response as an independent task.

Provide confidence intervals for paired differences on the same tasks. Large margins and consistent direction across seeds are more persuasive than one selected best checkpoint. Declare checkpoint-selection rules before examining held-out test scores.

For teacher metrics, state how undefined all-success/all-failure prompts are handled and report valid counts. For diversity, report parse rates and correctness-conditioned diversity, not only all-output variation.

Preserve evidence needed for debugging

Retain a sample of aligned teacher/student records, peer IDs, reward labels, termination causes, probability normalization metadata, and service versions. This lets a reviewer trace an unexpected improvement or collapse to its actual inputs.

A divergence curve without scoring metadata cannot distinguish learning from a changed temperature. A success curve without evaluator identity cannot distinguish a better model from a more permissive parser.

The standard for a conclusion

A defensible conclusion names the tested model, domain set, budget, metric, and uncertainty. It says which claim is supported and which remains open.

“Peer context improved LCB mean@8 in this fixed-budget run” is assessable. “Multi-rollout learning solves reasoning” is not. “MT integration recovered most specialist headroom” is meaningful when the anchors and aggregation are supplied.

Source notes

Known settings are from MT-MOPD's training/evaluation appendices and MR-MOPD's setup and appendices. Missing-detail identification and the reproduction program are the book's audit and recommendations.

Previous: 21. An Implementation Blueprint with Explicit Assumptions · Contents · Next: 23. Limits, Guarantees, and What Small KL Cannot Promise

23. Limits, Guarantees, and What Small KL Cannot Promise

Distillation is an agreement objective. Its practical value depends on agreement with the right supervisor at the right states. The papers show useful empirical results, but neither provides a general theorem that all specialist capabilities can be retained or that peer-conditioned learning converges to correctness.

This chapter separates mathematical implications from broader claims. It also identifies experiments that could turn plausible explanations into stronger evidence.

Agreement can be bounded without proving correctness

For normalized distributions, Pinsker's inequality relates total variation and KL:

At a fixed context, small KL therefore bounds differences in probabilities assigned to token events. It does not say that the teacher's preferred token is correct.

If the teacher places high probability on a wrong continuation, matching it faithfully preserves that weakness. A correctness argument needs a premise about teacher performance in addition to an agreement bound.

Local smallness is not automatically sequence smallness

For fixed-length trajectories of $T$ model decisions and a fixed teacher policy with compatible support, the KL chain rule gives

If the average conditional KL is at most $\epsilon$, then sequence KL is at most $T\epsilon$. Pinsker then gives

with the trivial upper bound one also applying. For any fixed binary success event, its probability difference is bounded by that total variation.

At $T=1{,}000$ and $\epsilon=0.001$, this upper bound is about 0.707. It is loose but illustrates that a small per-token average can accumulate over a long trajectory.

This derivation assumes a coherent fixed teacher sequence distribution, fixed length, and complete conditional KL. A corrected partial top-$k$ loss does not supply the same bound. Target-specific privileged contexts and random length normalization need additional treatment.

Measured average KL may conceal important states

A response contains many syntax and connective tokens. They can dominate a token average. A few decisive tool calls or reasoning branches may have much larger mismatch.

An average over visited states also leaves unvisited states unmeasured. If the student never reaches a useful branch, a low measured divergence elsewhere does not establish that it can follow that branch.

Outcome-stratified and decision-point diagnostics can help, but they require an operational definition of meaningful decisions. A hand-selected set can introduce bias. Report the selection rule.

Capacity limits capability integration

A finite student may not represent all teachers' conditional distributions simultaneously. Even identical architectures do not make independent specialist updates mutually compatible. Each specialist uses the model's capacity differently.

Routing reduces irrelevant target conflict, but shared parameters still need to support all prompted behaviors. A smaller student adds a stronger representational constraint.

The most useful empirical question is how recovery changes with student capacity, teacher diversity, number of domains, and domain mixture. The MT study demonstrates a few settings rather than a full phase diagram.

Teachers can disagree about legitimate behavior

Two teachers can prefer different correct proof styles or tool strategies. A likelihood loss may treat one as less likely even if both pass the evaluator. This can narrow stylistic or strategic diversity without changing task correctness.

Conversely, broader lexical diversity does not guarantee a wider repertoire of valid strategies. The MR diversity analysis illustrates the need to separate representation-level variation from correctness-conditioned algorithmic variety.

An improved objective might account for multiple valid outcomes or verifier feedback. That changes the supervision problem and requires new experiments.

Verifier errors enter through different paths

In MT, flawed specialist rewards can produce flawed teacher behavior. The integrated student can inherit that behavior even if integration itself is numerically stable.

In MR, flawed labels directly change which peers are presented as successes and failures. A mislabeled near-correct program can become an example to avoid. A mislabeled exploit can become a positive reference.

Use independently checked evaluation where possible. Training and testing with the same weak verifier can overstate useful progress. In open-ended tasks, reliable success labels may be difficult to define at all.

Context privilege has a deployment boundary

A teacher with successful peers sees information the student will not receive during a first attempt. Distillation can absorb general patterns but may not reproduce all instance-specific advantages.

The relevant evaluation is the final student's intended deployment behavior. If deployment also uses peers, that is a different inference procedure and should have its extra search cost reported.

The papers' training-time advantages do not imply a requirement to run peer sampling or specialist ensembles at every user request.

Stability is a property of a recipe and regime

Same-origin teachers work well in the tested MT setup. One external replacement fails. This supports compatibility-aware supervision but does not identify a necessary-and-sufficient origin condition.

Useful follow-ups include external-teacher alignment interventions, matched-sharpness comparisons, target-temperature studies, and learning-rate retuning. These would distinguish distribution mismatch from a particular optimization sensitivity.

For MR, useful follow-ups include robust peer selection under noisy labels, error-type balancing, and adaptive context budgets. An adaptive policy should be compared at matched compute and report how it changes the visited training distribution.

What convergence would require

A theorem would need assumptions about the student family, teacher support, optimization, data coverage, and evolving state distributions. If the teacher also changes or receives target-dependent peer context, the target is more complex than one fixed distribution.

Local loss descent does not guarantee global task-reward improvement. A gradient estimator can reduce divergence on sampled contexts while a shared-parameter update worsens another domain or changes future state occupancy.

The empirical method can still be effective without a theorem. Precision about the missing assumptions prevents practical success from being inflated into an unsupported guarantee.

Research questions that follow from the evidence

How many domains can be integrated before recovery falls? Which mismatch diagnostic best predicts transfer failure? Can teacher targets be conditioned on verified failures without increasing label-noise sensitivity? Does a full occupancy-gradient objective improve long-horizon behavior over local semi-gradients?

How should compound tasks be routed? Can sparse losses preserve rare but important tool actions? What is the best budget for teacher development versus integration? Does repeated-round improvement survive when all domains and general capabilities are evaluated?

These questions are natural extensions of the papers. Their answers remain open in the supplied evidence.

Source notes

The empirical limitations are grounded in MT-MOPD and MR-MOPD's limitations appendix. Pinsker and chain-rule calculations are standard mathematical background applied under stated assumptions. Research directions are proposals.

Previous: 22. Designing a Reproduction That Answers the Right Question · Contents · Next: 24. Synthesis and Frequently Asked Questions

24. Synthesis and Frequently Asked Questions

The two MOPD methods improve different parts of on-policy supervision. Multi-Teacher MOPD selects an appropriate domain specialist. Multi-Rollout MOPD gives the teacher instance-specific evidence from verified attempts. Their shared premise is that the supervision actually delivered on student states matters more than a teacher's name or size alone.

This chapter collects questions a reader may still have after learning the equations and experiments. Answers refer to the specified paper versions, and possible extensions are labeled as such.

Are the two papers describing one algorithm?

No. They share an acronym and an on-policy foundation. MT-MOPD integrates separately trained domain teachers. MR-MOPD conditions a teacher on successful and failed peers from one prompt. The book's prefixes distinguish them.

Does “multi-teacher” mean averaging all teacher logits?

The reported method routes each prompt to its domain teacher. Averaging logits, averaging probabilities, or minimizing several teacher KLs at the same state defines other targets. Those alternatives are not interchangeable with prompt routing.

Does MOPD require a teacher at inference time?

The final distilled student can generate responses by itself. Specialist and peer-conditioned teachers are training resources. An inference procedure that continues to call teachers or generate peer groups is a separate deployment design with additional cost.

Is the student necessarily smaller?

No. MT's primary purpose is consolidating specialist capabilities, and its same-origin controlled teachers share the student's architecture. MR also tests larger-teacher-to-smaller-student transfer. Distillation can compress models, but compression is not the definition of either MOPD contribution.

Why not just use the best teacher?

“Best” depends on the task. A math specialist may not retain the strongest repository-repair behavior. MT aims to put domain improvements into one model. If one teacher already excels everywhere and serves at acceptable cost, the integration problem may be different.

How can teachers teach on responses they would never generate?

Teacher forcing conditions them on supplied student histories. A causal teacher assigns next-token probabilities after those histories even if they are unlikely under its free-running policy. Whether its recovery predictions are useful is an empirical question.

Must the student rollout be correct?

No. Incorrect attempts can be valuable because they expose the student's actual states and mistakes. MT can score them without a success label during integration. MR needs verified peer labels to construct its intended context, but a target itself can be a failure.

Does on-policy training eliminate all distribution shift?

It addresses the difference between teacher-generated training prefixes and student-generated prefixes. It does not eliminate changes in prompts, tools, decoding, model versions, context lengths, or future policy states. Asynchronous lag can also create a mismatch within training.

Is the log difference a real environment advantage?

It is an advantage-like local supervision quantity. $\log q(y_t)-\log p(y_t)$ says whether the teacher assigns more conditional probability to the sampled token. It is not a measured improvement in task success or a complete return-to-go.

Why can the sampled-token loss be negative?

The detached-advantage surrogate is a signed training loss, not a nonnegative KL value. Its scalar value should not be interpreted as divergence. Full KL and the corrected sparse loss have different diagnostic meanings.

Are the immediate PG equations exact?

They have a clean fixed-context reverse-KL derivation under current-policy sampling. The complete on-policy objective also depends on which trajectories the student visits. Treating sampled prefixes as fixed gives a local semi-gradient. Clipping and stale samples add further approximations.

Why add $-p+q$ to top-k KL?

A naive restricted term $p\log(p/q)$ has coordinatewise stationary point $p=q/e$ when mass can move outside the selected set. The correction makes its derivative $\log(p/q)$ and its minimum occur at equality on selected coordinates.

Does corrected top-k equal full KL?

Only when all vocabulary coordinates are included does the correction sum cancel and recover full reverse KL. With a subset, the objective leaves individual tail probabilities unscored. It fixes a truncation pathology without becoming an unbiased full-vocabulary estimator.

Is top-k distillation the same as top-k decoding?

No. One chooses which teacher probabilities enter the loss; the other restricts the tokens that can be generated. MT's reported evaluation uses no top-$k$ decoding cutoff even though its alternative training loss uses teacher top-64 targets.

Why did a larger teacher hurt?

The tested external math teacher had a larger initial divergence from the student, and the runs showed deterioration and entropy/stability problems. This supports compatibility-aware teacher selection. It does not prove that every external or larger teacher is unsuitable after appropriate adaptation.

What does 0.9373 mean?

It is the average normalized recovery of specialist-over-SFT headroom across three domains. It is not 93.73% raw benchmark accuracy, not a fraction of all human abilities, and not a universal capability-retention rate.

Did MT win every benchmark?

No. It won the reported normalized aggregate and had a balanced domain profile. Other integration methods scored higher on some individual columns. Its teacher row also represents separate specialists, so the comparison is about consolidation.

Why can the student exceed a teacher on a benchmark?

The student has prior knowledge, receives many updates, and is not an exact global copy. Cross-domain effects and sampling noise can contribute. In MR, the supervising teacher also sees privileged context. A numerical gain on one benchmark does not imply unrestricted superiority.

Is MR merely copying successful answers?

The student is trained on its own prefixes against a peer-conditioned distribution. Successful peers are evidence for the teacher, not necessarily the target sequence. Failed peers add explicit negative context. The objective is therefore different from offline SFT on passing responses.

Did earlier self-distillation already use successful rollouts?

Yes. The original SDPO paper describes using successful rollouts as implicit feedback in scalar-reward environments. MR's contribution should be read as structured peer-context enrichment, particularly the success-failure contrast, rather than inventing the general use of successful attempts. SDPO paper.

Why can failures help without being useful alone?

A failure often needs a positive reference to reveal the relevant contrast. Without it, the teacher sees what went wrong but may not identify a valid replacement. The MR ablation supports complementary value from positives and negatives, with exceptions across individual metrics.

What happens if all attempts fail?

The desired positive-peer context is unavailable. A real implementation needs a specified fallback, such as no-peer scoring, failure-only context, resampling, or external references. The manuscript does not completely specify all such cases, so this book does not invent a definitive recipe.

Can more peers ever hurt?

Yes, more text can consume context, add redundancy, increase scoring cost, or distract the teacher. The reported all-peer LCB settings do not beat the compact mixed context. Chemistry has a higher longer-run all-peer result, so there is no universal best count established.

Do the signal metrics prove causal error localization?

They show associations between teacher likelihood and verifier outcomes under changed contexts. Undefined prompt cases and the Brier mapping need care. Selected case studies illustrate plausible localization but do not quantify population-wide causal effects.

Does greater diversity imply better reasoning?

Not by itself. Lexical variety and AST node-type variety can increase without adding correct algorithms. Evaluate correctness-conditioned diversity, parse rates, and strategy distinctions if those are the intended claims.

Is teacher scoring free because latency is hidden?

No. Overlap can hide wall-clock delay while teacher accelerators still perform work. Total cost also includes specialist development, environment execution, target storage, and network traffic. Report marginal and end-to-end cost separately.

Can the methods be combined?

A domain teacher could score a target with peer evidence, producing $q_d(\cdot\mid x,C_i,y_{<t})$. That is a plausible proposed combination. Neither supplied paper validates that joint method or justifies adding their separate gains numerically.

What should a reader remember?

Treat supervision as a conditional system: who teaches, what the teacher sees, which states are visited, how distributions are compared, and how success is measured. The two papers offer useful changes to that system, while leaving important questions about compatibility, coverage, cost, and generalization.

Source notes

Answers summarize the method and evidence analyzed throughout the book from MT-MOPD and MR-MOPD. The SDPO comparison is checked against its primary abstract. Interpretive answers preserve the qualifications developed in earlier chapters.

Previous: 23. Limits, Guarantees, and What Small KL Cannot Promise · Contents · Next: 25. Notation, Glossary, and an Auditable Source Map

25. Notation, Glossary, and an Auditable Source Map

This chapter provides a reference for readers returning to a specific equation, experiment, or source claim. The book studies fixed arXiv versions rather than treating paper titles as moving targets. Local textual source snapshots accompany the chapters so the reported values can be checked against the retrieved TeX.

Primary papers and versions

MT: Wenhan Ma and coauthors, MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training, arXiv:2606.30406v1, submitted June 29, 2026. Abstract, HTML, source download.

MR: Weichen Yu and coauthors, Multi-Rollout On-Policy Distillation via Peer Successes and Failures, arXiv:2605.12652v2, revised June 1, 2026; original submission May 12, 2026. Abstract, HTML, source download.

Sources were retrieved for this book on October 13, 2026. The local _sources/manifest.json records archive and retained-file SHA-256 hashes. Source files remain attributed research material; the surrounding explanations, derivations, examples, and proposed designs are authored for this book.

Local source organization

Paths below are relative to src/content/docs/papers/mopd/:

book/                         authored Markdown chapters
_sources/manifest.json        retrieval metadata and checksums
_sources/multi-teacher/
    xiaomi.tex                top-level inclusion order
    chapters/                 method, experiments, appendices
    latex_fig/                active numerical tables and figure wrappers
    custom.bib, references.bib
_sources/multi-rollout/
    main.tex                  top-level inclusion order
    text/                     active sections, appendix, case studies
    custom.bib, custom_new.bib

This is a textual audit snapshot, not a complete compilable paper package. Figure PDFs, logos, and class/style files are not retained. Some unused text files are retained as received; an inclusion in the archive is not evidence that it appears in the compiled paper.

For MT, xiaomi.tex determines active chapter order. For MR, main.tex includes intro, related_work_short, prelim, method, exp_1, exp, exp_3, conclusion, and appendix. The appendix includes additional active files. Comments, \iffalse blocks, and the first argument of revision macros are not active prose.

This distinction matters because the MR archive contains older numbers and alternative descriptions that should not override the active v2 manuscript.

Symbols