Brief
inspect_evals v0.14.3: SWE-bench runs that miss a marker now error instead of scoring 0.0
The inspect_evals v0.14.3 release reworks how SWE-bench Verified results are graded and how failed runs are classified, plus smaller changes to three other evals. Evidence here is the release notes alone; no external testing of the claims is available.
The v0.14.3 release of inspect_evals reworks how several evals report outcomes. The most consequential change is in SWE-bench Verified (v5-C): grading now runs through the SWE-bench harness (swebench.harness.grading.get_eval_report), and run outcomes are classified using SWE-bench's START_TEST_OUTPUT and END_TEST_OUTPUT markers. Runs that reach both markers are graded normally — resolved maps to correct, otherwise incorrect. Runs missing a marker are treated as infrastructure failures and now raise, so the sample is marked ERRORED and stays eligible for --retry-on-error instead of being recorded as a misleading 0.0.
The same release adds a grade_parse parameter to Coconot (v3-B), a multi-turn confirm_to_complete solver to TAC (v6-C), pinned reasoning effort and larger token limits on TAC (v5-C), and a sandbox_config parameter on 3CB (v4-B).
Our reading
For anyone running agent evaluations, the old behaviour turned a broken container or killed test process into what looked like a model failure, quietly dragging an average down. Marking those samples ERRORED changes both the retry path and the aggregate numbers, so scores produced before and after this change are not obviously comparable. Teams comparing harness versions, or relying on --retry-on…
What to do or watch
Re-run SWE-bench Verified under v0.14.3 with --retry-on-error before comparing against any earlier score, and treat a spike in ERRORED samples as an infrastructure signal rather than a model result. For 3CB, note that non-docker providers require a custom spec passed via sandbox_config.
Source details and supporting facts
Each line is stated by the page named above it.
Stated by github.com
- SWE-bench Verified (v5-C) reworked its scorer to grade via the SWE-bench harness (swebench.harness.grading.get_eval_report).
- Runs that reach both START_TEST_OUTPUT and END_TEST_OUTPUT markers are graded normally (resolved means correct, otherwise incorrect).
- Runs missing a marker are treated as infrastructure failures and now raise, so the sample is marked ERRORED and stays eligible for --retry-on-error rather than being scored as a misleading 0.0.
- Coconot (v3-B) adds a grade_parse task parameter, where 'strict' is the default and 'paper' keeps the original reference parser for faithfulness.
- TAC (v6-C) adds a multi-turn confirm_to_complete solver that injects a neutral user confirmation when a model stalls without booking, up to 2 times.
- TAC (v5-C) pins reasoning_effort='medium' on tac and tac_welfare and raises max_tokens from 4096 to 16384.
- 3CB (v4-B) added a sandbox_config parameter; passing a per-sample function returning a SandboxEnvironmentSpec runs the eval against a non-default provider, and non-docker providers require a custom spec via sandbox_config.
Sources
- UK AI Safety Institute (inspect_evals releases)Text stored 16 September 2026
How this story was checked. Written from the 1 page listed above, stored 16 September 2026; claims checked against that stored text on 16 September 2026.
What that means
- 7 of 7 reported statements were confirmed against the page that carries them; the rest were removed rather than published.
- Figures in the text were required to appear in the stored source text: yes. Identifiers: yes.
- The check reads stored text only: no claim rests on a fresh look that did not happen.
- Where the reporting was silent, the text says so instead of filling the gap.