papers.luk.media

Scaling Properties of Same-Family On-Policy Distillation

arXiv:2609.32722 · Paper published 2026-09-26 · Original paper ↗

By measuring student learning progress through divergence from initialization, this paper fits joint power laws that predict a student language model's peak test accuracy and learning speed from model scale and teacher accuracy before distillation begins.

Listen · 7 minOpen audio
Problem
On-policy distillation trains a student language model using token rewards from a reinforcement-learning teacher. However, developers cannot predict how much capability transfers across model sizes. Evaluating pairings currently requires expensive, trial-and-error training across every parameter combination.

How it works

  1. Gather model scale and accuracy inputs The system inputs student scale, teacher scale, and teacher accuracy. Subtracting teacher accuracy from 100 percent calculates remaining teacher error, establishing baseline capacity limits before training.
  2. Cap effective teacher scale The system caps effective teacher scale at student parameter count. If a teacher exceeds student size, its effective scale equals student size, reflecting observed diminishing returns from oversized teachers.
  3. Calculate peak student accuracy Inputs enter a multiplicative power law that multiplies student scale, capped teacher scale, and remaining teacher error. This outputs predicted peak student error, showing accuracy improves with student scale and smaller teacher scale at equal teacher performance.
  4. Project early learning speed A second power law calculates initial transfer rate, measuring accuracy gain per unit of divergence from initialization. This projects how fast the student absorbs knowledge during early training.

Example from the paper

Consider distilling mathematical reasoning into a 14 billion parameter student base model using a 0.5 billion parameter expert teacher scoring 39.8 percent accuracy. The power law inputs 14B student scale, 0.5B effective teacher scale, and 60.2 percent teacher error, forecasting peak accuracy near 77.0 percent. In the paper's actual experiment, the student reached a measured peak accuracy of 77.7 percent, confirming the law accurately predicts weak-to-strong capability transfer before training begins.

Evidence

Reported evidence and its scope
Reported resultTest conditionsWhat it supports
Joint power laws predict held-out 14B student peak accuracy within 0.7 percentage points.Outer-scale extrapolation holding out 14B student models across 25 teacher-student pairs (0.5B to 14B) on math benchmarks.Supports forecasting peak student accuracy before training using model scale and teacher score.
Off-policy fine-tuning on teacher text before distillation reduced 14B student peak accuracy from 77.7% to 58.3%.Comparison between pure on-policy distillation and an off-policy cold start using a 0.5B expert supervising a 14B student.Shows off-policy cold starts impair weak-to-strong transfer by forcing the student into the weak teacher's distribution.
A 4-step bootstrapped chain (0.5B -> 1.5B -> 3B -> 7B -> 14B) reached 76.8% peak accuracy, below direct distillation's 77.7%.Comparison of sequential intermediate distillation chains against direct single-step on-policy distillation.Demonstrates intermediate teachers bottleneck latent features, proving higher teacher score does not guarantee better teaching.

Limits

  • Power laws were established empirically on Qwen2.5 models (0.5B to 14B) on math reasoning with rollouts up to 2,000 tokens; applicability to other tasks is unproven.
  • Beyond a divergence threshold near d_transfer of 0.30, distillation leaves the linear regime and enters a noisy phase of saturation or accuracy loss.
  • Effective teacher scale saturates at student parameter count; scaling a teacher beyond student size yields diminishing returns.
Terms used here
On-policy distillation (OPD)
A training method where a student model generates response tokens and receives reward feedback scored by a teacher model.
Useful-transfer regime
The initial training phase where student accuracy increases linearly with the square root of token divergence from initialization.
Delta-OPD
A distillation variant where token rewards equal the teacher's post-RL log probabilities minus its pre-RL base log probabilities.
Token-mean reverse KL divergence
A sample-based estimator measuring average log-probability differences per token between student updates and baseline initialization.
Visual summaryVisual summary of Scaling Properties of Same-Family On-Policy Distillation
Source and authors

Paper authors: Yuntai Bao, Qinfeng Li, Guoqing Jiang, Liwei Chen, Zhiheng Qin, Xuanping Li, Wenqi Zhang, Xuhong Zhang

arXiv:2609.32722 ↗ · Episode released 2026-10-11 · Explanation updated 2026-10-13