An agent that passed 77.4% of runs passed all five on only 53% of tasks. IBM's new diagnostic targets that gap.

The number the post leads with is a gap, not a score. A ReAct agent using GPT-4.1 on AppWorld's test_normal split succeeded on 77.4% of runs across five repetitions — but it succeeded in all five runs for only 53.0% of tasks, a 24.4-point consistency gap. The authors say that on hard tasks the gap reaches 30 points, and they frame the stakes for mission-critical work such as reconciling a financial transaction or checking a contract for an obligation, where a workflow that succeeded once may fail the next time a user makes the same request.

The post separates two metrics. Mean@k runs a benchmark k times and averages the pass rate; it answers how good the agent is on average and is what a leaderboard figure like "77% accurate" usually means. Pass^k is the fraction of tasks where the agent succeeds on all k runs, and it answers the question a real user has — will this exact request work again? The authors state plainly that Pass^k is not Pass@k.

The Consistency Analyzer is the diagnostic. It resamples an agent's own recorded trajectory to find flip-prone decision points: steps where the model was one token-sample away from doing something different. The post says it needs one trace and no ground truth, and that it resamples each decision point in that trace with a single call requesting k completions (k=5 by default) rather than re-running the task end to end.

Consistency guidelines are a new guideline type in altk-evolve, built on top of that analyzer and sitting alongside the earlier ALTK-Evolve approach, which distills an agent's past trajectories into reusable guidelines injected at inference time. The post reports that turning the diagnosis into guidelines halves the gap, from 24.4pp to 12.0pp, with same-task Pass⁵ up 16.0pp and similar-task Pass⁵ up 13.0pp, and that this costs nothing in average accuracy. It also states that the guidelines generalize rather than patching a single trajectory.

For anyone shipping a tool-calling agent, the consequence is that an average success rate is a statement about the fleet of attempts, not a guarantee for the attempt in front of a user. The post's own method is cheap to imitate at small scale: take a task your agent handles, run the same request five times, and count how often all five runs pass — that count is the Pass^k number that averages hide.