Brief
Inspect Evals v0.14.4 fixes scorer grade-binding and setup behavior in four evaluation suites
A point release of the UK AI Safety Institute's inspect_evals package corrects scoring and setup behavior in four evaluations; the release notes describe no new evaluations.
XSTest (v3-A) carries the clearest consequence for reported numbers: the grader grade-pattern now binds to the model grader's final GRADE: verdict instead of the first GRADE: token in its step-by-step reasoning. The scorer previously passed a custom leftmost-matching pattern; it now relies on Inspect's DEFAULT_GRADE_PATTERN, which greedily binds to the final grade.
Three further fixes ship together. AgentHarm (v2-B) fixes realtime log serialization when AgentHarm scorers use a semantic judge model. GDM Dangerous Capabilities: Self-reasoning (v4-A) and AssistantBench (v4-A) move the system prompt and required tool selection into setup so they remain applied when callers override the solver.
Our reading
For anyone rerunning these suites or comparing scores across package versions, the XSTest change is the one that can shift results, because it corrects a scorer rather than a model. Teams using a semantic judge for AgentHarm logs, or overriding the solver in the other two suites, should expect behavior that matches the documented setup instead of the caller's override.
What to do or watch
Re-run XSTest under v0.14.4 and record the package version alongside any score you publish, since results produced under the earlier leftmost-matching pattern may not be directly comparable.
Source details and supporting facts
Each line is stated by the page named above it.
Stated by github.com
- XSTest (v3-A) fixes the grader grade-pattern to bind to the model grader's final GRADE: verdict instead of the first GRADE: token in its step-by-step reasoning.
- The XSTest scorer previously passed a custom leftmost-matching pattern and now relies on Inspect's DEFAULT_GRADE_PATTERN, which greedily binds to the final grade.
- AgentHarm: Harmfulness Potential in AI Agents (v2-B) fixes realtime log serialization when AgentHarm scorers use a semantic judge model.
- GDM Dangerous Capabilities: Self-reasoning (v4-A) moves the system prompt and required tool selection into setup so they remain applied when callers override the solver.
- AssistantBench (v4-A) moves system prompts into setup for the closed-book and web-search tasks so they remain applied when callers override the solver.
Sources
- UK AI Safety Institute (inspect_evals releases)Text stored 17 September 2026
How this story was checked. Written from the 1 page listed above, stored 17 September 2026; claims checked against that stored text on 17 September 2026.
What that means
- 5 of 5 reported statements were confirmed against the page that carries them; the rest were removed rather than published.
- Figures in the text were required to appear in the stored source text: yes. Identifiers: yes.
- The check reads stored text only: no claim rests on a fresh look that did not happen.
- Where the reporting was silent, the text says so instead of filling the gap.