When Do Stale Rollouts Preserve On-Policy Prompt Rankings?

Estimating current-policy prompt utility under $\pi$ from fixed rollouts generated by an earlier policy $\beta$ introduces score error. Let $\Delta_k$ be the current top-$k$ margin and $e$ the maximum error over prompts. If $\Delta_k>2e$, the stale and on-policy rankings select exactly the same set. In the other direction, correcting only the history distribution or only the action value does not generally identify the current ranking. Small KL and high ESS cannot recover information that was omitted.

RLVR systems reuse behavior-policy responses to reduce rollout cost. But stable optimization on stale groups and preservation of the current-policy prompt order are different claims. Our empirical estimand is projected cosine alignment with a held-out validation gradient, not downstream benchmark reward. The consumed output is the top-$k$ decision induced by the score, not the scalar score itself.

For response $Y=(A_1,\dots,A_T)$, terminal reward $R(Y)$, and token score $z_t=\nabla_\theta\log\pi(A_t\mid H_t)$, define token ratios $r_j=\pi(A_j\mid H_j)/\beta(A_j\mid H_j)$, prefix products $P_t=\prod_{j<t}r_j$, and suffix products $S_t=\prod_{j>t}r_j$. They produce four population estimators.

$g_{10}$ and $g_{01}$ remove different sources of bias, so neither is a generic substitute for the other. The equality for $g_{11}$ assumes finite horizon, support, and exact unclipped ratios. It does not automatically certify a practical capped estimator.

Order the current scalar utilities as $s_{(1)}\ge\cdots\ge s_{(n)}$ and denote stale estimates by $\widetilde s_i$. Define the selection margin and uniform error by

After perturbation, every true top-$k$ item remains at least $\Delta_k-2e>0$ above every unselected item. The proposition applies to arbitrary scalar scores. It does not imply that the empirical cosine proxy's $e$ is known without on-policy evidence.

For any selector that observes only history-corrected $g_{10}$ or only action-value-corrected $g_{01}$ information, there are two binary-reward prompt pools with the same observation law and different true top-$k$ sets. The construction holds at arbitrarily small $D_{KL}(\pi\Vert\beta)$, unit normalized ESS, and after GRPO group normalization for every $K\ge2$. The minimax exact-selection error is therefore at least $1/2$.

We divide candidate and validation on-policy samples into ranking split R and independent reference halves A/B. R constructs the fresh comparator. The mean A/B scalar score is the held-out utility reference, while overlap between the A- and B-induced top-$k$ sets measures split-half reliability. Independent fixed tie streams prevent accidental tie agreement from entering the reliability estimate.

Under a Gaussian latent-score model, population split-half reliability determines a ceiling on the expected precision that an independent selector can demonstrate against the noisy reference. This result assumes Gaussianity, equal noise variance, and conditional independence; it is not a distribution-free ceiling. Exact certification also has gap-dependent sample cost $\Omega(\sigma^2\Delta^{-2}\log(1/\delta))$.

Positive checkpoints share one continuous adapter-and-optimizer lineage. Each fresh group is used for one optimizer epoch, so the PPO-form ratio is one at the only loss evaluation up to numerical error. clip_epsilon=0.2 is recorded but is not claimed as an effective trust region. Homogeneous-reward groups are valid zero-advantage groups and are counted rather than retried.

A comparison is admissible only if the one-sided 95% lower bound on split-half reliability is at least $2k/n$ and the lower bound on fresh selection gain is positive. Effective requires at least 80% of registered seeds to pass both gates, have positive stale-gain lower bounds, and retain at least 50% of fresh gain at the lower bound. Ineffective requires at least 80% to pass the gates but have nonpositive stale-gain upper bounds. Everything else is inconclusive.

Three 4×H100 nodes run the same command against one physically shared volume. A shared queue assigns an entire dataset-seed family to one worker and preserves its 0→25→100→400 lineage. Rollouts bind exact policy/adapter hashes and RNG domains. Interrupted generation resumes after the last complete prompt group; training resumes from the newest complete five-step checkpoint. Compute mode disables Hugging Face credential discovery and networking, then checks local snapshot revisions, content hashes, schemas, and verifier self-tests before GPU work.

The aggregate table remains frozen until all 40 points pass objective, lineage, generation, analysis, and manifest contracts and finish 10,000 bootstrap replicates. Partial cluster output is not a paper result.

This tests observed-stratum replication outside the main OLMo-3 math/code setting. Architecture, capacity, domain, and verifier all broaden, but architecture and capacity vary jointly and cannot be causally separated. Separate single-domain runs do not establish cross-domain transfer, mixed-domain curricula, alternative-objective robustness, full-parameter robustness, multilingual transfer, or universal generalization.

The earlier positive-only response cross-entropy sweep was SFT, not RLVR. Its 20 runs remain only as a supervised policy-shift ablation in the appendix. They cannot resume a GRPO checkpoint, populate the main regime table, or support an RLVR result. LoRA specifies trainable parameters; it does not define the learning objective.

The theory concerns exact unclipped population ratios and an unnormalized directional gradient. The experiment uses capped ratios, CountSketch, and cosine normalization, so it is not a direct implementation of the theorem. The main study covers one raw 7B model, two domains, LoRA, and one-epoch GRPO. Bootstrap intervals are conditional on the fixed prompt pool and stale selector. Downstream reward after training on the selected subset is not yet the primary measured outcome.

The current publishable contribution is theory plus a prespecified protocol. We report no estimator winner, drift boundary, or model/domain generalization result until the complete 40-point main matrix and 72-point extension pass their artifact gates.