The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation
arXiv:2609.36484 · Paper published 2026-09-29 · Original paper ↗
RIDE pushes a student model beyond its reinforcement-learning teacher by training student hidden representations to overshoot teacher representations along the layerwise residual vector between the teacher and its pre-learning base model, bypassing noisy output-token sampling.
Standard on-policy distillation matches student token outputs to a reinforcement-learning teacher, capping student performance at the teacher's level. Extrapolating beyond the teacher in output space fails because the language-model head suppresses 79.8% of internal representation residual energy into low-gain logit directions, and next-token sampling introduces advantage variance that scales quadratically with (lambda - 1)^2.
How it works
Generate text prefixes. A prompt enters this step. The student model generates a response sequence to create a text prefix. The student, frozen teacher, and frozen pre-learning base model process this exact same prefix to produce layerwise hidden states—the model's internal numerical description of that context. Evaluating all models on identical text ensures that state comparisons isolate reinforcement-learning updates rather than trajectory divergences.
Calculate internal model changes. Hidden states from the teacher and base model enter this step. At each layer and position, the base state vector h_B is subtracted from the teacher state vector h_T to yield the residual Delta = h_T - h_B. This isolates the specific directional modification carved into internal representations by reinforcement learning before passing through the final classification head.
Project target internal states. The teacher state h_T, residual Delta, and extrapolation multiplier lambda > 1 enter this step. Target state h* = h_T + (lambda - 1) * Delta is constructed by extending beyond the teacher along the learning direction. Because h* is computed directly from model forward passes on a fixed prefix, it eliminates next-token-resampling noise conditional on that prefix.
Update student parameters. Student states h_theta and projected targets h* enter this step. Student parameters theta are updated by minimizing the squared distance between h_theta and h* across supervised layers and response tokens, scaled by lambda^(-2). This shifts student intermediate representations beyond the teacher along the reinforcement-learning direction.
Illustrative example
Consider a single layer and token position where the pre-learning base hidden state is base = 1.0, the reinforcement-learning teacher state is teacher = 3.0, and the extrapolation multiplier is lambda = 1.25. The learning change vector is change = teacher - base = 3.0 - 1.0 = 2.0. RIDE calculates the target state as target = teacher + (lambda - 1) * change = 3.0 + (1.25 - 1.0) * 2.0 = 3.5. The student representation is trained toward target = 3.5, overshooting the teacher's state of 3.0 along the learning direction.
Evidence
Reported evidence and its scope
Reported result
Test conditions
What it supports
RIDE achieved higher average reasoning accuracy than all teacher-matching and output-extrapolation baselines, scoring 56.38% on R1-Distill-1.5B (vs 55.30% teacher, 54.50% hidden-state matching OPRD, 49.87% output extrapolation ExOPD) and 66.07% on Qwen3-4B (vs 65.59% teacher, 62.01% OPRD, 48.75% ExOPD).
Mean Avg@16 accuracy over 3 independent training seeds across AIME24, AIME25, and AIMO benchmarks using DAPO-Math-17K prompts for 500 steps with lambda = 1.25.
Supports that hidden-state extrapolation allows students to exceed RL teachers, whereas output-space extrapolation degrades performance below standard teacher matching.
Output-space advantage variance rose 12.6-fold as the extrapolation factor increased from 1.0 to 2.0, causing validation format score to drop from 80.9% to 64.4%, whereas RIDE maintained zero next-token conditional sampling variance across all multipliers.
Diagnostic evaluation on 9,152 fixed on-policy validation prefixes from the R1-Distill-1.5B model pair.
Confirms that output-space extrapolation amplifies next-token sampling noise quadratically by (lambda - 1)^2, while hidden-state regression stays deterministic for a fixed prefix context.
Replacing RIDE's directional vector with a random Gaussian vector (55.04%), reversed vector (53.90%), mismatched base model origin (54.75%), or trajectory-mismatched residual (55.12%) all scored lower than RIDE with the true reinforcement-learning residual (56.38%).
Ablation experiments on R1-Distill-1.5B at lambda = 1.25, holding the displacement magnitude constant while altering residual direction.
Demonstrates that student gains depend specifically on following the internal direction established by reinforcement learning on identical context prefixes.
Limits
Shared Representation Space: The student, teacher, and pre-learning base model must share the exact same neural network architecture, vocabulary tokenizer, and initialization checkpoint.
Uniform Global Scaling: RIDE applies a single scalar extrapolation multiplier lambda across all layers and token positions, which does not adapt to layer-specific optimal scaling rates.
Unverified Calibration and Safety: Supervising student hidden states toward artificial targets beyond actual model outputs means student output calibration and safety alignment are not guaranteed to match the teacher.
Terms used here
On-Policy Distillation
A training process where a student language model generates text sequences and receives learning signals based on those specific generated token prefixes.
Language-Model Head Attenuation
The suppression of internal representation changes by the final classification layer, which directs 79.8% of internal residual energy into low-gain output directions.
RL-Induced Representation Residual
The layerwise vector difference Delta = h_T - h_B between an RL-trained teacher's hidden states and its pre-RL base model's hidden states on identical text prefixes.
Conditional Sampling Variance
The fluctuation in an optimization gradient or advantage estimate caused strictly by resampling the next token given a fixed prompt context.
Visual summarySource and authors
Paper authors: Hao Li, MeiJia Chen, Weijie Ren, Donghan Li, Zijun Tian, Jingchun Huang, Naibo Wang
arXiv:2609.36484 ↗ · Episode released 2026-10-11 · Explanation updated 2026-10-13
RealCompanion introduces a longitudinal benchmark derived from 27,218 real-world human-companion messages across up to 120 days, enabling systematic evaluation of an AI companion's selective memory retrieval, profile fact extraction, trait inferencing, and restraint when history is unnecessary.
2026-10-01 · Evaluation and benchmarks · AI agents · Language models
By measuring student learning progress through divergence from initialization, this paper fits joint power laws that predict a student language model's peak test accuracy and learning speed from model scale and teacher accuracy before distillation begins.
2026-09-26 · Language models · Reinforcement learning · Efficient AI
Raven packages language models with domain tools into reusable worker units, using a central host agent to break user requests into workflow graphs, validate file-based handoffs before execution, and run subtasks in parallel.
2026-09-27 · AI agents · Evaluation and benchmarks · Language models
papers.luk.media