Double-Sample N=16 (No Retry)

The critical causal control. 16 first-attempt solutions without any retry gives only +1.4pp, vs +10.1pp from a single retry. This 7x gap proves the failure signal causes a distribution shift, not just additional sampling.

Hypothesis

If retry works merely by generating more samples (importance sampling), then generating MANY samples without retry should achieve equivalent gains. We test: 16 independent first-attempt solutions per failed problem, filter correct, SFT. This provides 16x more draws than single retry.

If retry = more sampling: N=16 should match or exceed single retry.

If retry = distribution shift: N=16 will be much worse (all from same distribution).

Method

1. Identify frontier: Generate one solution per problem. Identify all failures.
2. Generate N=16: For each frontier problem, generate 16 INDEPENDENT first-attempt solutions (no retry prompt, no failure signal).
3. Filter: Keep only solutions arriving at the correct answer.
4. SFT: Fine-tune on filtered correct solutions.

Key difference from retry: the model has NO knowledge it previously failed. Each solution is from a fresh first-attempt prompt. This is pass@16 sampling.

Configuration

Results

Method	Samples/problem	Delta	Compute vs baseline
Retry (STaR, seed 42)	1 after failure	+10.1pp	2x
Double-sample N=16	16 (no failure signal)	+1.4pp	16x
Temp-uniform N=8	8 (no failure signal)	+2.6pp	8x
Rejection sampling (all)	8 (all problems)	-2.0pp	8x

Why N=16 Fails Despite More Compute

1. No distribution shift: All 16 samples come from the same first-attempt distribution. Correct solutions on frontier problems are drawn from the model's existing mode, not alternative reasoning paths.
2. Redundant training data: The correct solutions from N=16 are statistically similar to what the model already produces when it succeeds. SFT on these does not teach new strategies.
3. No mode suppression: Without failure notification, the model's dominant (incorrect) solution strategy is not suppressed. It keeps generating the same type of approach 16 times.

Training Curves

At: /data/ughai-sandbox/opsd_experiments/double_sample_n16/. Loss drops normally but eval barely moves. The model fits training data without learning transferable improvements because training data is too similar to existing behavior.

Interpretation

This experiment resolves WHY retry works:

• Ruled out: Retry = more sampling (importance sampling on frontier)
• Confirmed: Retry = qualitative distribution shift from failure awareness

The failure signal: (1) suppresses the dominant incorrect strategy, (2) activates alternative reasoning from pre-training, (3) produces genuinely novel solutions for SFT.

Connection to Other Experiments