Graded Verifier vs Binary Filter

Partial credit (graded verifier, weighted SFT) gives +4.2pp, while strict binary filtering gives +12.9pp. The binary filter is 3x better. Near-miss solutions actively dilute the training signal.

Hypothesis

Intuition suggests that "near-miss" solutions (those that get most steps right but fail at the final answer) should be valuable training data. A graded verifier that assigns partial credit should outperform a harsh binary filter that discards everything except perfectly correct solutions.

Expected: Graded >= Binary (more data should help, partial credit captures useful reasoning).

Actual: Binary is 3x better. Partial credit is noise, not signal.

Method

Binary Filter (standard STaR)

1. Generate retry solutions for failed problems
2. Check final answer against gold: PASS or FAIL (no middle ground)
3. Keep only PASS solutions
4. SFT with uniform weight on all kept solutions

Graded Verifier (partial credit)

1. Generate retry solutions for failed problems
2. Score each solution on a 0-1 scale: (a) final answer correct = 1.0, (b) intermediate steps correct but final wrong = 0.5-0.8, (c) approach correct but execution wrong = 0.3-0.5, (d) completely wrong = 0.0
3. Keep solutions scoring above 0.3 (includes near-misses)
4. SFT with loss weighted by the score (higher-scoring solutions get more weight)

Configuration

Results

Condition	Training examples	Baseline	Post-training	Delta
Binary filter	~3,200 (only correct)	40.1%	53.0%	+12.9pp
Graded verifier	~5,500 (includes near-miss)	40.1%	44.3%	+4.2pp
Gap				3x worse

Why Partial Credit Hurts

• Subtle error propagation: A solution that gets 5/6 steps right but makes one arithmetic error teaches the model that flawed reasoning is acceptable. The error is hidden in an otherwise correct-looking trace.
• Signal dilution: Mixing verified-correct solutions with near-miss solutions reduces the average quality of the training set. The model's SFT gradient points less clearly toward "correct reasoning."
• False confidence: Near-miss solutions often arrive at a wrong answer via plausible-looking reasoning. The model learns these plausible-but-wrong patterns, potentially increasing confidence in incorrect approaches.

Training Curves

Binary: loss drops cleanly, eval improves steadily. Graded: loss drops faster (more data) but eval improvement stalls after step 200. The model fits the mixed-quality data but does not generalize as well.

Logs at: /data/ughai-sandbox/opsd_experiments/graded_vs_binary/

Interpretation

This result has a clear practical implication: always use the harshest possible filter. In any self-improvement pipeline:

• If you have a binary verifier (answer match, type-checker, test suite): use it. Accept only perfect correctness.
• Never use process reward models (PRMs) or partial-credit heuristics to "save" near-miss solutions.
• The apparent waste (discarding 40% of generated solutions) is actually signal concentration.

This is Level 4 of the mechanism hierarchy: graded verification is actively harmful compared to binary filtering.

Connection to Other Experiments