Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability
arXiv:2610.08448 · Paper published 2026-10-06 · Original paper ↗
Restricting cross-tokenizer on-policy distillation to a student-selected top-16 subset of shared vocabulary at strictly aligned token positions eliminates noisy mismatch supervision while retaining over 96% of full shared-vocabulary accuracy gains across heterogeneous model pairs.
Previous cross-tokenizer on-policy distillation approaches maximize alignment coverage by using mean squared error span losses or token merging on mismatch groups where token boundaries differ between teacher and student. However, supervising these mismatched spans introduces weakly aligned or conflicting gradient signals whose relative magnitude grows during training. Consequently, expanding supervision coverage over unaligned token groups degrades downstream mathematical and coding accuracy compared to strict matching.
How it works
Align token boundaries. The student generates a text response y given prompt x. Both student policy pi_theta and frozen teacher pi_T tokenize y into token sequences, and shared character offsets partition the response into aligned token groups. Groups containing exactly one student token and one teacher token form strict 1:1 positions A_1:1(y), while multi-token mismatch groups A_mis(y) are discarded to prevent conflicting gradient noise.
Filter shared vocabulary. At each strict position r in A_1:1(y), both models predict next-token probability distributions over their respective vocabularies V_theta and V_T. The method restricts and renormalizes both probability distributions over the shared vocabulary intersection V_cap = V_theta cap V_T. This isolates common vocabulary entries, enabling direct probability comparisons without needing cross-tokenizer projection matrices.
Select top predictions. For each strict position r, the student identifies the top-k entries (e.g., k=16) in V_cap that receive its highest predicted probabilities. Both student and teacher distributions are truncated to this top-k subset K_k^(r) and renormalized to sum to 1. This isolates dense probability mass where teacher and student overlap, filtering out low-probability vocabulary noise.
Optimize reverse KL loss. Reverse Kullback-Leibler divergence KL(pi_theta || pi_T) is calculated over the top-k support K_k^(r) at each strict position r in A_1:1(y). Student parameters theta are updated by gradient descent on loss L_1:1(theta). This aligns student predictions on visited prefixes without introducing destabilizing gradients from mismatched token spans.
Illustrative example
Suppose a student tokenizes the number '25' as a single token while the teacher tokenizes it as two tokens '2' and '5'. Previous methods apply a log-probability span loss across this mismatch group, producing gradients with near-zero directional agreement (cosine near 0) relative to strict token gradients. The proposed method discards this '25' span. At a subsequent strictly aligned token 'plus' shared in V_cap, the student selects its top-16 predictions, renormalizes both models' probabilities over those 16 tokens, and updates student parameters via reverse KL divergence. This ensures parameter updates rely strictly on clean, reliable teacher signals.
Evidence
Reported evidence and its scope
Reported result
Test conditions
What it supports
Strict top-16 distillation achieved 32.64% Full accuracy for Qwen2.5-7B-Instruct to Llama-3.2-3B-Instruct, retaining 96.3% of Strict full's 32.86% score.
On-policy distillation on 20,000 DAPO-Math and CodeForces prompts; evaluated on 7 math benchmarks (mean@32) and 3 code benchmarks (mean@8).
Supports that a student-selected top-16 vocabulary subset retains nearly all distillation gains; does not establish performance in off-policy training.
Adding span MSE loss (lambda = 1.5) reduced Full accuracy across all three model pairs (Qwen->Llama: 32.86% to 31.84%; Granite->Phi: 42.49% to 41.69%; Granite->Qwen: 47.35% to 46.55%).
Sweep of span log-probability MSE loss weight lambda in [0, 1.5] added to strict reverse KL loss across three heterogeneous teacher-student pairs.
Demonstrates that maximizing structural alignment coverage via span MSE hurts downstream accuracy; does not prove all possible span objectives fail.
Strict top-16 surpassed all evaluated baselines (SimCT, GOLD, Extended ULD, ULD) across all three pairs in Full accuracy (e.g., Granite->Phi reached 42.26% vs SimCT 41.68% and GOLD 41.75%).
Comparison against cross-tokenizer distillation baselines implemented in KDFlow under identical on-policy rollout and evaluation conditions.
Supports that prioritizing supervision reliability over alignment coverage improves distillation performance over prior cross-tokenizer methods.
Limits
Requires a non-empty shared vocabulary intersection V_cap = V_theta cap V_T; models with disjoint token vocabularies cannot perform direct shared-vocabulary distribution matching.
Supervision is restricted to 1:1 strictly aligned token positions; sequences with high tokenization divergence drop unaligned tokens (though empirical strict coverage was 85.57%–96.98% across tested pairs).
Adding span log-probability MSE supervision introduces gradient noise, as span gradient norms grow up to 28.0x relative to strict gradients while maintaining near-zero directional agreement (cosine near 0).
Evaluated specifically on on-policy distillation using reverse KL divergence on reasoning and code tasks; findings do not directly prove results for off-policy distillation or preference tuning.
Terms used here
On-Policy Distillation (OPD)
A distillation method where the student model generates its own response sequences and receives teacher probability feedback on the exact prefixes it visits.
Strict 1:1 Position
A sequence location where exactly one student token and one teacher token span the identical character offset in the generated text.
Student-Selected Top-k Subset
The k shared vocabulary entries receiving the highest predicted probabilities from the student model at a given position.
Supervision Reliability
The property that teacher feedback targets at aligned positions provide non-conflicting, effective gradient direction for student learning.
AgentGarten couples physics engines to a shared neural renderer through untextured 3D geometry maps. This separation lets AI agents interact with photorealistic virtual worlds in real time while keeping physical rules and object coordinates explicitly controllable in code.
2026-10-08 · AI agents · Computer vision · Efficient AI
RealCompanion introduces a longitudinal benchmark derived from 27,218 real-world human-companion messages across up to 120 days, enabling systematic evaluation of an AI companion's selective memory retrieval, profile fact extraction, trait inferencing, and restraint when history is unnecessary.
2026-10-01 · Evaluation and benchmarks · AI agents · Language models
RIDE pushes a student model beyond its reinforcement-learning teacher by training student hidden representations to overshoot teacher representations along the layerwise residual vector between the teacher and its pre-learning base model, bypassing noisy output-token sampling.
2026-09-29 · Reinforcement learning · Language models
papers.luk.media