PI Distillation Research Dashboard

4-Level Mechanism Hierarchy

Multi-Seed Confidence Intervals

Cross-Domain Summary

Research Timeline: How Understanding Evolved

All Experiment Results (Sorted by Gain)

The Mechanism: Failure-Aware Distribution Shift

The double-sample control (+1.4pp) vs retry (+10.1pp) proves that the failure signal genuinely shifts what the model generates. It is not more sampling; it is qualitatively different reasoning.

What Does NOT Matter (PI Content Ablations)

What DOES Matter

Per-Level Analysis: Gains Concentrate at the Frontier

OPSD gains are concentrated at Level 3-4 (the frontier band, 36-46% baseline solve rate). Levels that are too easy or too hard show minimal/no improvement. This is the inverted-U operating AT THE PROBLEM LEVEL.

Key insight: Gains concentrate at the frontier (30-50% solve rate per level). The inverted-U is not just a model-level phenomenon; it operates per-problem within a single model.

Three-Step Mechanistic Explanation

1. Suppresses the dominant mode. The model's first attempt reflects its highest-probability solution path. "You failed" tells it that path is wrong, forcing exploration of lower-probability alternatives.
2. Activates self-correction heuristics. Pre-training on human text includes patterns of "I made an error, let me reconsider..." that retry prompts activate. These encode approach-switching and deeper checking.
3. Contracts the search space. Rather than re-exploring the full solution space (as N=16 i.i.d. does), the model eliminates its default approach, concentrating on alternatives. This is more efficient than blind resampling.

Lean: Complete 5-Method Hierarchy (MiniF2F, Kimina-1.5B)

Lean theorem proving is the domain exception. The complete hierarchy shows that ONLY structural PI (have-skeleton OPSD) produces positive gains. All retry-based methods are negative, and combining skeleton with retry interferes rather than compounds.

Key insight: Skeleton+retry (-2.0pp) is WORSE than skeleton alone (+3.3pp). The retry mechanism actively destroys the benefit of structural PI. In constrained domains, retry forces the model to abandon its skeleton-guided strategy, exploring alternatives that violate the type-checker. The mechanisms are antagonistic, not additive.

Worst case: Lean heuristic 3-iter reaches -3.6pp (progressive degradation). Iterative retry is poison for formal domains.

Predictions and Status

Practitioner Recipe

1. Generate first-attempt solutions (N=1 per problem)
2. Verify against oracle (answer match, type-checker, test suite)
3. Retry failures with "try again carefully" (any prompt works)
4. Filter to ONLY correct retries (binary, no partial credit)
5. SFT on (problem, correct_retry) pairs (LoRA, 500 steps)
6. Iterate if desired (diminishing returns expected)

Expected gain: +8-12pp at 1.7B scale on math, +23pp on code (if baseline > 40%). Exception: formal verification (Lean) needs structural PI.

The Inverted-U: Baseline Competence vs Gain

Retry gains are maximized when the model has moderate baseline competence (30-60%). Too easy (nothing to retry) or too hard (retries also fail) both yield zero gain.

The sweet spot appears to be 30-55% baseline accuracy for math/code domains. Below ~10%, the model cannot produce correct retries. Above ~65%, there is nothing left to learn from retry.

8B LR Sweep Results

The 8B null result was a hyperparameter artifact. The optimal LR shifts upward with model scale, likely because LoRA updates need larger steps to overcome the model's stronger priors.

32B Result: Inverted-U Confirmed

The 32B result (+0.6pp) confirms the inverted-U hypothesis but for a different reason than expected. The issue is NOT LoRA capacity or LR tuning; it is that 32B's baseline (68.3%) is already past the sweet spot. There is nothing left to learn from retry at this performance level.

Implication: Retry is a method for STRUGGLING models (30-55% baseline). At 32B, the model is too good for retry to help on MATH-500. It might still help on harder benchmarks (AIME, putnam-level) where 32B baseline is lower.

Retry-Count Inverted-U (K sweep)

Increasing retries per problem shows diminishing and then negative returns. K=3 is optimal; K=5 regresses, likely due to overfitting on low-quality late retries.

Key insight: Retry count has its own inverted-U. More retries initially help (more chances to find correct solutions) but past K=3, late-retry solutions are low quality (the model has exhausted its good alternative strategies) and training on them dilutes the signal.

Currently Running (3 jobs, Day 3 steady-state 2026-05-11)

Cluster: CMH (us-east-2), p5en-queue. Completed Day 3: 8B full FT (+9.6pp verified), 32B retry (+0.6pp), iterative R3 (-1.6pp), Lean type-checker PI (-1.2pp), Lean failure-diagnosis (-1.2pp), Lean heuristic 3-iter (-3.6pp), HAR math, skeleton-guided retry, skeleton+retry (-2.0pp).