MLOps

Pass^k: measuring AI agent consistency before production

Pass^k, the agent-consistency metric: the 24-point gap IBM measured on AppWorld, the ALTK-Evolve method, and what a k-run acceptance test changes for your agents.

Pass^k: measuring AI agent consistency before production

Pass^k: measuring AI agent consistency before production

An agent that completes a task once is not a reliable agent. On 15 September 2026 IBM Research published, on the Hugging Face blog, a study that puts a number on something every team running AI agents has felt: a ReAct agent built on GPT-4.1 succeeds in 77.4% of runs on the AppWorld benchmark, yet succeeds on all five repetitions of the same task only 53.0% of the time. A 24.4-point gap between average performance and guaranteed performance. Alongside it, IBM ships an open-source tool, ALTK-Evolve, that halves that gap without changing the model.

For an MLOps team this is not academic. It is the difference between an agent you can put in front of users and an agent that "works in the demo". This post explains the Pass^k metric, IBM's diagnostic method, and what it changes in how an agent should be evaluated before going live.

The context

An AI agent chains decisions: which tool to call, with which arguments, when to stop. Each decision is a draw from a probability distribution over tokens. Even at temperature 0 that distribution is not always decisive: when two options are tied or nearly tied, a tiny platform-level perturbation (batching, floating-point arithmetic, a server version) is enough to flip the decision. On a twenty-step task, one flip changes the outcome.

Leaderboards hide this because they publish an average. The average says how often the agent succeeds; it does not say whether it succeeds on the same task every time. The second question is the one an operations lead actually asks.

Three metrics, one that matters in production

IBM formalises the distinction with three measures, computed over k independent runs of each task:

  • Mean@k: the average pass rate across the k runs. The leaderboard metric.
  • Pass@k: a task counts as solved if at least one of the k runs succeeds. An optimistic measure, for scenarios where you can retry.
  • Pass^k: a task counts as solved only if all k runs succeed. The pessimistic measure, for a user who gets no second chance.

The ordering always holds: Pass^k ≤ Mean@k ≤ Pass@k. The authors call the difference between Mean@k and Pass^k the consistency gap. On AppWorld (168 tasks from the test_normal split, five runs per task), the GPT-4.1 agent shows Mean@5 = 77.4% and Pass^5 = 53.0%: a 24.4-point gap.

Why this is not Pass@k

Pass@k is the right tool when you can sample several answers and keep the best one, as in code generation checked by tests. A production agent has no such luxury: it runs once, with side effects (an email sent, an order placed). Pass^k is the metric that matches that reality.

IBM's method: diagnose without ground truth

ALTK-Evolve is an open-source system that turns an agent's own past trajectories into reusable guidelines, distilled automatically and injected back at inference time. The 15 September addition is a guideline type dedicated to consistency, produced in two stages.

1. The consistency analyzer

From a single recorded trajectory, the analyzer replays each decision step through controlled resampling: one extra model call per step, drawing k completions (five by default) and checking whether they agree. A step whose completions diverge is an unstable decision. The point that matters for an operations team: this detection needs no ground truth, no replay of the full task and no grader. It runs on production traces.

2. Guideline generation

Flagged unstable decisions become natural-language guidelines in the format of ALTK-Evolve's existing retrieval pipeline, injected into the agent's context on later runs. The agent learns, in effect, to commit where it used to hesitate.

The numbers

On AppWorld, with the GPT-4.1 ReAct agent and five runs per task:

  • Pass^5 rises from 53.0% to 69.0%;
  • Mean@5 rises from 77.4% to 81.0%: the average does not degrade, it improves;
  • the consistency gap narrows from 24.4 to 12.0 points;
  • medium-difficulty tasks gain 22.9 points of Pass^5 (+44% relative), hard tasks 14.3 points (+45%);
  • guidelines transfer to unseen similar tasks: +13.0 points.

On a weaker model, gpt-oss-120b, the gain exists but stays modest: Pass^5 goes from 10.1% to 16.1% on the same tasks (+6.0 points), +8.7 points on similar tasks. IBM draws a clear conclusion: consistency is orthogonal to capability. A bigger model does not fix the problem, and a small model cannot be "made consistent" beyond what it can do.

What this changes for AI teams

Change the acceptance metric. An agent should not go to production on a Mean@1 measured once. Acceptance should run each scenario k times (five is a good start) and publish Pass^k next to Mean@k. In the agent rollouts we run, that is exactly what the evaluation set is for: replayed scenarios, a Pass^k threshold per criticality level, and a report showing the consistency gap before and after every prompt or model change.

Expect the gap to grow with difficulty. IBM's figures show the gap widening on medium and hard tasks. An agent that looks reliable on simple cases can be unpredictable on the cases that justify its existence. Acceptance scenarios must cover long chains, not just simple questions.

Diagnose on production traces. The consistency analyzer works on a recorded trajectory, without a grader. Plugged into existing observability (call traces, tool arguments), it flags unstable decisions from real traffic, which can then be handled with guidelines rather than a model change. It is an improvement loop at marginal inference cost: one extra call per analysed step, offline.

Do not reach for a bigger model first. IBM's recommendation is explicit. Before raising the inference bill, measure the consistency gap, identify the unstable decisions and constrain them. It is cheaper, and often more effective, than a model swap that moves the problem without solving it.

Keep the average under watch. Any intervention on consistency must respect one constraint: mean accuracy must not drop at any difficulty level. IBM presents this as a hard constraint of the method. A guideline that forces a choice can also force a mistake; the evaluation set has to catch it.

Limits worth keeping in mind: the study covers one benchmark (AppWorld) and two models; gains on the weaker model are limited; and behaviour at higher values of k is not addressed. Pass^5 is not Pass^50.

Key takeaways

  • On AppWorld a GPT-4.1 agent succeeds in 77.4% of runs but on only 53.0% of tasks every time: a 24.4-point consistency gap.
  • Pass^k (success on all runs) is the metric that matches a production agent; Pass@k is an optimistic retry measure.
  • ALTK-Evolve detects unstable decisions by resampling a recorded trajectory, with no ground truth, and turns them into guidelines injected at inference.
  • Result: Pass^5 from 53.0% to 69.0%, Mean@5 from 77.4% to 81.0%, gap down to 12.0 points; modest gains on a weaker model.
  • Recommendations: publish Pass^k, expect a larger gap on hard tasks, diagnose before switching models.

Industrialising AI agents? SeedVision offers 3-5 day AI audits and 15-30 day production rollouts. See the packages or book a 30-min call.

Sources: Your Agent Aced the Task. Will It Do It Again? (IBM Research on the Hugging Face blog, 15 September 2026), the ALTK-Evolve repository and the technical report on arXiv.

Cover photo: Photo by Vishnu Mohanan on Unsplash.