Improves operational reliability for mission-critical automated workflows.
What did IBM Research find about AI agent consistency?
IBM Research measured what most benchmarks hide: whether an agent that succeeds once can succeed again. On AppWorld, a ReAct agent backed by GPT-4.1 posted a Mean@5 of 77.4%, and it passed all 5 repeated runs on only 53.0% of tasks, a 24.4-point consistency gap.
The team’s write-up puts the gap at 30 points on hard tasks, and the agent ran at temperature 0.0, so the variance survives the decoding settings most teams trust.
A workflow that succeeded in rehearsal can fail the next identical request, which turns a strong demo into a reliability problem the moment a customer depends on it.
Capability and consistency are different axes, and production runs on both.
What was actually measured in IBM’s agent study?
The evaluation ran on AppWorld test_normal, 168 tasks across 9 simulated apps, executed by a ReAct agent on GPT-4.1, with 5 fresh runs per task after guideline generation.
AppWorld builds each task from interactive coding and API calls, the same shape of work as production agent workflows.
Mean@k, the number on every leaderboard, averages the pass rate across k runs. Pass^k counts only the tasks an agent passes on every single run, which is the experience of a user who asks the same thing twice.
Pass^k, the metric nobody reports, is the one your customer experiences.
Does a bigger model fix the consistency gap?
No, and the research is direct about it: consistency is orthogonal to capability, so a stronger model raises Mean@k without narrowing the gap.
The fix runs the other direction. The technical report describes the Consistency Analyzer, which resamples each decision point in a recorded trace with a single call requesting k=5 completions, with no ground truth and no end-to-end re-run, and scores which steps sit one token away from flipping.
Flagged steps become consistency guidelines, and the open-source ALTK-Evolve toolkit ships the whole pipeline, from analysis to guideline generation.
A real guideline from the study shows the flavor: when counting checkbox markers in a note, anchor the regex to the line instead of counting substrings, because note titles repeat the marker in a legend line. In the team’s 2-minute demo, 5 parallel runs of the same task split 3 to 2 over the counting strategy, and all 5 agreed once the guideline rode along.
Turning diagnosis into guidelines halves the gap from 24.4 points to 12.0 without costing average accuracy.
What does the consistency gap change for teams shipping agents?
If an agent touches invoices, contracts, or customer records, a 53.0% repeat rate means about 1 in 2 tasks carries a hidden chance of a different outcome.
Teams that ship agents without measuring variance bake that coin flip into production, and the failure lands on whoever reads the output, not on the leaderboard the agent was sold from.
Agent reliability is one of the patterns we track every week in the signals archive, because every vendor demo this year is a single lucky run until someone repeats it.
Until you measure repeats, every agent score you quote is a rehearsal grade.
A quote bot sends the same renewal file twice on a Friday afternoon, once from the morning test and once from the live queue. Both runs read the same CRM, the same rate card, and the same customer record, and the two documents disagree on coverage by 2 line items.
That is the 53.0% number wearing office clothes. The agent averaged 77.4% across its benchmark runs, which got it approved, and it passed all 5 repeats on half the tasks at best, which is what your compliance team now owns.
The cheap fix is measurement: resample the decision points before rollout and watch which steps flip, instead of paying for a bigger model after the incident.
What should your team do about AI agent reliability?
Report Pass^k next to Mean@k on every agent you own, and start at k=3 if 5 runs strains the budget. The gap shows up at any k.
Run the Consistency Analyzer over recorded production traces, because it needs one trace per task, no grader, and one extra model call per decision step.
The measured results: aggregate Pass^5 rose from 53.0% to 69.0% while Mean@5 held at 81.0%, narrowing the gap from 24.4 points to 12.0. Medium tasks gained 22.9 points, hard tasks gained 14.3, and almost a third of inconsistent tasks became ones the agent passes every run.
The guidelines transfer, too. On similar tasks in the same scenario, Pass^5 still rose 13.0 points, and on the weaker gpt-oss-120b model the same-task gain was 6.0 points from a 10.1% baseline, which tells you the method captures reusable failure patterns rather than memorizing one trace.
Measure the repeat before you trust the run.
Source: Hugging Face (IBM Research)
