Same agent, reordered fields: task success falls from 85% to 19%
A new paper shows your agent harness — prompts, tools, even field order — carries baked-in preferences, and frozen-model scores mislead you when picking one.
Copy markdown
Field order is invisible to the task, not to your model
In MiniWoB++, reversing the order of observation fields leaves every optimal decision unchanged — yet a trained model's success collapses from 85.0% to 19.2%. Post-training bakes the exact shape of your harness into the policy, so a cosmetic prompt change can quietly wreck a deployed agent.
You're probably comparing harnesses wrong
The paper splits an agent's return into a harness 'ceiling' (the best it allows) minus the model's 'mismatch' (return lost to bad choices). A frozen model can score worse on the better harness — so A/B-testing scaffolds with one fixed model misleads. Measure ceiling-minus-mismatch instead.
It flips which coding harness you'd pick
They test five real coding scaffolds — DeepAgents, Pi, DeepSeek Harness, Terminus-2 and mini-SWE-agent — on Terminal-Bench Lite with a frozen Qwen3.5-9B. Ceiling-minus-mismatch reproduces the accuracy ranking; either term alone doesn't. The better harness is often the one a frozen model looks worse on, until you fine-tune under it.
The diagnostic is open, MIT, runnable now
The HRP code is on GitHub under MIT: six MiniWoB++ harnesses with exact policy evaluation and oracle ceilings, plus the Terminal-Bench Lite coding study (needs Docker, vLLM 0.28, NVIDIA GPUs). Point it at your own harness to size the headroom you're leaving on the table before you spend a training run.