DefenseBio / Research
Evaluating coding agents on physiological model development.
Higher is better. Overall Elo pools matched attempts across all three tasks. Valid predictors beat failed attempts; two failures skip. No confidence intervals estimated. Both Granite 4.0 Micro rows are self-hosted on RunPod: the untuned base model and a GRPO-tuned checkpoint (step 28). They ran the same suite and judge configuration.
| Model | Prediction Elo | Valid attempts |
|---|---|---|
| GPT-6 Astramedium effort | 1268.4 | 8/9 |
| Claude Opus 5.5medium effort | 1238.9 | 8/9 |
| Claude Sonnet 5.5medium effort | 1200.2 | 6/9 |
| Gemini 4 Argonmedium effort | 1133.0 | 5/9 |
| Gemini 3.8 Flashmedium effort | 1077.0 | 4/9 |
| Kimi K3high effort | 929.8 | 2/9 |
| Granite 4.0 Micro + GRPOstep 28 effort | 863.1 | 1/9 |
| DeepSeek V4 Flash 0423high effort | 767.3 | 0/9 |
| Granite 4.0 Microbase effort | 765.1 | 0/9 |
| Grok 4.7medium effort | 757.1 | 0/9 |
Heart-rate estimation
Core-temperature forecasting
Metabolic-cost prediction
Shared scale; three trials per task. Valid predictors beat failed attempts; two failures skip. No confidence intervals estimated.
65 saved reviews · 7 missing · GPT-6.1 Sol, Medium. Scores assess research decisions separately from prediction Elo.
Episode-weighted means over observable scores; nulls are excluded. The judge received compressed decision and development evidence, not full code or hidden test results.
The retry succeeded, but its explanation named a different error from the one in the evidence.
The retry succeeds after failure, but its stated diagnosis does not match the supplied failure; the actual repair is not visible.Read verdict and cited events ↗
After a regularization change worsened development MAE, the model kept its original candidate.
Explicitly carries baseline feedback into the next trial and retains the better baseline after the alternative worsens MAE.Read verdict and cited events ↗
A proposed CV and physics-feature experiment returned the same development loss as the baseline.
A follow-up experiment follows the baseline result, but does not explicitly interpret it; identical reported losses receive no visible diagnosis.Read verdict and cited events ↗
The model correctly selected its valid development candidate; the final predictor still failed. Process and prediction scores capture different outcomes.
Explicitly selects the only valid evaluated artifact and cites both results, appropriately excluding the failed candidate.Read verdict and cited events ↗
Agents train two predictors, evaluate on development data, and select one for hidden replay. Person-balanced error determines prediction Elo; a separate judge assesses the experiment decisions.
This retrospective pilot produced 34 valid predictors from 90 attempts, with 83 of 90 process reviews available. Failed attempts remain in the matched outcomes. Pulse ECG references are provisional; fixed cohorts and three agent seeds do not establish population-level ranking confidence.
