01Introduction
This pilot tests whether a coding agent can construct an executable physiological predictor, use development feedback, and select a final artifact. The model being compared is the coding agent. Its output is a separate trained predictor.
Ten agents each receive three tasks at seeds 0, 1 and 2. These repeats use the same people; they are not independent biological cohorts or evidence of clinical performance.
02The environment
Pulse uses GalaxyPPG wrist PPG and acceleration with provisional ECG-derived labels, split across 13/5/5 training/development/test people. Heat forecasts measured rectal temperature at +5/+10/+20 minutes in a 24-person cycling cohort, split 14/5/5. Load predicts gross metabolic watts at 4 km/h and 0/5/10/15 kg load in EVOBREATH, split 28/10/10.
Training labels are public. Development labels and all test recordings stay with the trusted controller. Predictor programs have no network or GPU access. Whole people remain within one split.
03Methodology
Each episode schedules two open-ended Python training programs, development evaluation, then final selection. Each of the three participant phases has a 1,536-token output limit including reasoning and at most one response repair. The complete participant request body is capped at 4,096 bytes. Model API waits overlap; Docker workers run one at a time under the frozen CPU/memory allocation.
Six participants use Medium effort. Kimi K3 and DeepSeek V4 Flash use High because their native endpoints did not advertise Medium. Economy provider routes and quantization differ and are recorded in the public configuration.
Lower person-balanced MAE wins task matches. A valid attempt beats an invalid attempt; two invalid attempts skip. Overall Elo pools all matched task-and-seed comparisons, so partial results contribute and every participant retains a numeric rating, including models with no valid predictors. This post-run analysis replaces the original complete-suite ranking in the site; the original suite scores remain in the downloadable data. Elo starts at 1,000 with K=32 and is relative to this opponent pool.
The separate GPT-6.1 Sol Medium judge reviews compressed experiment decisions, development results and selection evidence. It assigns five 0–4 criteria or null when unobservable, citing event IDs. It receives neither the competitor identity nor hidden test scores, and its packet does not include the full executable code. Seven reviews are missing. Browse the saved verdicts and cited evidence ↗
04Results
All 90 attempts sealed. 34 produced valid predictors. GPT-6 Astra leads the pooled-task Elo table, followed by Sonnet 5.5 and Opus 5.5. Ratings reflect both predictive accuracy and execution failures; all ten models receive a score.
Fine-tuned Granite Micro for Defense (3B) is a notable feasibility result under the fixed evaluation budget: it produced one valid predictor in nine attempts and scored 863.1 pooled Elo. That ranks 7th of 10, above three competitors but below the six models with higher ratings. It is about 98 Elo above the untuned Granite Micro baseline, which produced no valid predictors. This is encouraging evidence that a small fine-tuned model can complete part of this workflow; one successful attempt is too little to establish a reliable advantage over the base model or a competitive result across tasks.
Higher is better. Overall Elo pools matched attempts across all three tasks. Valid predictors beat failed attempts; two failures skip. No confidence intervals estimated. Both Granite Micro runs are self-hosted on RunPod and use the same suite and judge configuration.
View exact scores and coverage
| Model | Prediction Elo | Valid attempts |
|---|---|---|
| GPT-6 Astramedium effort | 1268.4 | 8/9 |
| Claude Opus 5.5medium effort | 1238.9 | 8/9 |
| Claude Sonnet 5.5medium effort | 1200.2 | 6/9 |
| Gemini 4 Argonmedium effort | 1133.0 | 5/9 |
| Gemini 3.8 Flashmedium effort | 1077.0 | 4/9 |
| Kimi K3high effort | 929.8 | 2/9 |
| Fine-tuned Granite Micro for Defensestep 28 effort | 863.1 | 1/9 |
| DeepSeek V4 Flash 0423high effort | 767.3 | 0/9 |
| Granite 4.0 Micro | 765.1 | 0/9 |
| Grok 4.7medium effort | 757.1 | 0/9 |
Pulse
Heart-rate estimation
- GPT-6 Astra11411141.5 Elo; 3 of 3 valid attempts; medium effort.
- Sonnet 5.511411141.2 Elo; 2 of 3 valid attempts; medium effort.
- Opus 5.510761076.0 Elo; 2 of 3 valid attempts; medium effort.
- Gemini 3.8 Flash10531053.1 Elo; 1 of 3 valid attempts; medium effort.
- Fine-tuned Granite Micro for Defense999998.8 Elo; 1 of 3 valid attempts; step 28 effort.
- Gemini 4 Argon987986.8 Elo; 1 of 3 valid attempts; medium effort.
- Kimi K3975975.4 Elo; 1 of 3 valid attempts; high effort.
- Granite 4.0 Micro880880.5 Elo; 0 of 3 valid attempts; base effort.
- DeepSeek V4 Flash875875.0 Elo; 0 of 3 valid attempts; high effort.
- Grok 4.7872871.6 Elo; 0 of 3 valid attempts; medium effort.
Heat
Core-temperature forecasting
- GPT-6 Astra11891188.5 Elo; 3 of 3 valid attempts; medium effort.
- Sonnet 5.511811180.7 Elo; 3 of 3 valid attempts; medium effort.
- Opus 5.511561155.6 Elo; 3 of 3 valid attempts; medium effort.
- Gemini 4 Argon10561056.0 Elo; 2 of 3 valid attempts; medium effort.
- Gemini 3.8 Flash977976.9 Elo; 1 of 3 valid attempts; medium effort.
- Kimi K3972972.3 Elo; 1 of 3 valid attempts; high effort.
- DeepSeek V4 Flash869869.1 Elo; 0 of 3 valid attempts; high effort.
- Granite 4.0 Micro867867.4 Elo; 0 of 3 valid attempts; base effort.
- Grok 4.7867867.3 Elo; 0 of 3 valid attempts; medium effort.
- Fine-tuned Granite Micro for Defense866866.2 Elo; 0 of 3 valid attempts; step 28 effort.
Load
Metabolic-cost prediction
- Opus 5.511961195.7 Elo; 3 of 3 valid attempts; medium effort.
- Gemini 3.8 Flash11131112.5 Elo; 2 of 3 valid attempts; medium effort.
- GPT-6 Astra11121112.4 Elo; 2 of 3 valid attempts; medium effort.
- Gemini 4 Argon11061105.8 Elo; 2 of 3 valid attempts; medium effort.
- Sonnet 5.510151014.5 Elo; 1 of 3 valid attempts; medium effort.
- Fine-tuned Granite Micro for Defense895895.4 Elo; 0 of 3 valid attempts; step 28 effort.
- Kimi K3892892.0 Elo; 0 of 3 valid attempts; high effort.
- DeepSeek V4 Flash892891.7 Elo; 0 of 3 valid attempts; high effort.
- Grok 4.7891890.8 Elo; 0 of 3 valid attempts; medium effort.
- Granite 4.0 Micro889889.2 Elo; 0 of 3 valid attempts; base effort.
Shared scale; three trials per task. Valid predictors beat failed attempts; two failures skip. No confidence intervals estimated.
LLM judge verdicts
Browse all episode reviews ↗65 saved reviews · 7 missing · GPT-6.1 Sol, Medium. Scores assess research decisions separately from prediction Elo.
Episode-weighted means over observable scores; nulls are excluded. The judge received compressed decision and development evidence, not full code or hidden test results.
Successful repair, unsupported diagnosis
The retry succeeded, but its explanation named a different error from the one in the evidence.
The retry succeeds after failure, but its stated diagnosis does not match the supplied failure; the actual repair is not visible.Read verdict and cited events ↗
Retaining the better baseline
After a regularization change worsened development MAE, the model kept its original candidate.
Explicitly carries baseline feedback into the next trial and retains the better baseline after the alternative worsens MAE.Read verdict and cited events ↗
Ambitious proposal, identical results
A proposed CV and physics-feature experiment returned the same development loss as the baseline.
A follow-up experiment follows the baseline result, but does not explicitly interpret it; identical reported losses receive no visible diagnosis.Read verdict and cited events ↗
Sound selection, failed final attempt
The model correctly selected its valid development candidate; the final predictor still failed. Process and prediction scores capture different outcomes.
Explicitly selects the only valid evaluated artifact and cites both results, appropriately excluding the failed candidate.Read verdict and cited events ↗
05Failure analysis
56 attempts failed under this short protocol. Failure includes inability to return a valid tool response within the fixed attempt budget or produce an executable predictor. These are not zero-error scores and do not establish unconstrained model incapability. The downloadable results include a failure reason for every invalid attempt.
- Response/tool protocol: 45
- Other execution failure: 11
Infrastructure interruptions also occurred: Docker became unavailable, coordinators restarted from saved state, and the xAI endpoint returned 72,797 completion tokens despite a 1,536-token limit. That response was rejected and its charge retained. The request-cap-based cost estimate consequently stopped being a completion guarantee. These exceptions are documented with the reference protocol.
06Limitations
This is a retrospective, builder-exposed pilot with small fixed cohorts and provisional pulse reference quality. Three agent seeds do not support population-level ranking confidence. Tight generation limits, native tool compatibility, provider quantization and incomplete review coverage affect the results. Process judgments have not been validated against human expert ratings.
Confirmed provider charges are $1.66. Another $6.14 is held as conservative accounting for calls with unconfirmed billing; it is not confirmed spend. Neither amount is presented as a per-model cost metric.
Exact protocol and source snapshot ↗ · Evaluate a fine-tuned checkpoint ↗ · Download public results