AWS has shipped a new capability inside Bedrock AgentCore: AgentCore Evaluations, a framework-agnostic way to score AI agents running in production, regardless of whether they were built with LangGraph, the OpenAI Agents SDK, LlamaIndex, Google ADK, the Claude Agent SDK, or Strands Agents. The announcement, published August 26, 2026 on the AWS Machine Learning blog, addresses a problem most teams running more than one agent eventually hit: every framework historically ships its own evaluation tooling, and none of them talk to each other, which makes it painful to compare quality across a mixed fleet of agents or roll up a single dashboard for leadership.
If your team is past the single-agent POC stage and now operates agents built on different stacks — a common outcome once more than one team starts shipping — this release is worth a closer look.
Why evaluation gets hard at scale
In production, an AI agent isn't a model you query once: it's a chain of steps — reasoning, tool calls, model calls, user feedback — that can silently drift week over week. Measuring quality means capturing that full chain (a "trace") and scoring it against criteria like whether the agent achieved its goal, whether its answer was factually correct, and whether it actually helped the user.
The recurring pain point: every agent framework emits traces in its own proprietary shape. That has historically forced teams to either rebuild evaluation tooling every time they adopt a new framework, or standardize the whole org on a single stack — rarely realistic once a company grows through acquisitions, autonomous teams, or simply because different use cases call for different tools.
This isn't a new problem in the abstract — it's the same one OpenTelemetry solved for conventional application telemetry a few years back. Before wide adoption, every APM vendor shipped its own instrumentation agent, and switching monitoring tools often meant re-instrumenting the whole codebase. The ecosystem eventually converged on a shared trace format, which let dozens of tools offer different views over the same underlying data. AWS is betting the same dynamic can play out for agentic observability — starting from OpenTelemetry, already proven elsewhere, instead of asking an entire ecosystem to adopt a brand-new homegrown format.
A common substrate: OpenTelemetry
AgentCore Evaluations sidesteps this by building on OpenTelemetry, the open instrumentation standard already widely used for conventional application telemetry (traces, metrics, logs). Agent traces flow through the AWS Distro for OpenTelemetry (ADOT) into Amazon CloudWatch, where the service reads three span types:
- agent invocation spans — the top-level request: user prompt and final response;
- inference spans — model calls, including message history;
- tool execution spans — the tool name, its parameters, and its result.
To recognize these spans regardless of which framework produced them, the service accepts two semantic conventions in parallel: OpenTelemetry's GenAI conventions (the gen_ai.operation.name attribute) and the OpenInference specification (openinference.span.kind), automatically routing to the right handler based on the detected prefix. In practice: as long as your framework instruments its traces against either convention — which LangGraph, the OpenAI Agents SDK, LlamaIndex, Google ADK, the Claude Agent SDK, and Strands Agents already do — AgentCore Evaluations can score it with no extra work on your side.
The built-in evaluators
Three evaluators ship out of the box: GoalSuccessRate (did the agent complete the task's goal), Correctness (is the answer factually accurate), and Helpfulness (did the response actually address what the user needed). The service also supports custom LLM-as-a-judge evaluators for domain-specific criteria — useful if your team already runs a homegrown scoring rubric in production and doesn't want to start over.
Two evaluation modes for two different needs
AgentCore Evaluations ships two complementary modes, aimed at two different points in an agent's lifecycle:
- On-demand mode, built for CI/CD pipelines, with ground-truth comparison. This is the mode you'd run before a deploy, to check that a new prompt or logic change doesn't regress against a known test set — a non-regression suite for a system whose outputs aren't deterministic.
- Online mode, which continuously monitors live traffic at a configurable sampling rate. This is the mode that catches quality drift — an agent that quietly gets worse as data or usage patterns shift — without waiting for the next release cycle to notice.
What the announcement doesn't cover yet
Worth flagging before you evaluate this for adoption: the official post cites no customer examples and no performance numbers — no false-positive rate for the evaluators, no evaluation latency, no per-trace cost. This is a freshly documented technical capability, not yet a field-tested track record, so it's worth validating internally before betting tooling decisions on it.
Adopting it without re-instrumenting everything
The good news for a team already on one of the six supported frameworks: if OpenTelemetry instrumentation is already active at the framework level — which is the default for LangGraph, the OpenAI Agents SDK, LlamaIndex, Google ADK, the Claude Agent SDK, and Strands Agents, per the GenAI or OpenInference conventions — there's nothing to rewrite in application code to start sending traces. The real work sits upstream of that: making sure the ADOT pipeline is correctly configured to route those traces into CloudWatch, then deciding which evaluators to turn on and against what slice of traffic.
For a team running a homegrown agent that doesn't sit on any of these frameworks, the question is different: does your current instrumentation already follow one of the two supported conventions, or does it need adapting? That's the scenario with the highest adoption cost — but also the one with the clearest long-term payoff, since any OpenTelemetry-compliant observability tool, not just AgentCore Evaluations, will be able to read the same traces afterward.
What this means for AI engineering teams
If your team runs a single agent on a single framework, this is just one more option among the observability tools already on the market. Where AgentCore Evaluations actually changes the picture is for organizations running a mixed fleet — a customer-support agent in LangGraph, an internal research agent in LlamaIndex, a coding agent in the Claude Agent SDK, say. Without a shared evaluation substrate, comparing quality across those agents, or building one dashboard for a steering committee, means either rewriting tooling per framework or forcing the whole org onto one stack — rarely desirable when each team had good technical reasons for its choice.
AWS's bet with this evaluation/framework decoupling, using OpenTelemetry as the common language, is also a broader signal: agentic observability data is starting to standardize, echoing what happened to conventional application telemetry a few years back. Teams that instrument their agents properly today, with spans conforming to GenAI or OpenInference conventions, avoid a rebuild the day they want to plug in another evaluation tool — this one or a different one.
Concretely, for a team auditing or industrializing AI agents, the design checklist gains one useful line item: check, from the first lines of code, that trace instrumentation follows a standard convention rather than a homegrown format. That costs almost nothing to set up and avoids an expensive migration later, whatever evaluation or observability tool ends up winning out.
There's a second practical implication worth calling out: on-demand and online modes map to two different organizational needs, usually owned by different teams. CI/CD mode speaks to engineering teams who want a binary signal before merging a prompt or logic change. Online mode speaks to SRE/MLOps teams who want a drift signal over time, independent of any deploy. A service that covers both with the same evaluator definitions avoids the classic failure mode where CI says "green" and production drifts anyway, because the two sides were never scoring against the same criteria.
There's also a governance angle worth not underselling. Once an organization runs more than one agent, "which of our agents is the riskiest right now" becomes a legitimate question for a steering committee or a security team — and it's unanswerable if every agent is scored on different criteria, on different scales, by different tools. A shared evaluation substrate doesn't answer what quality bar is acceptable for which use case — that's still a human call — but it makes the comparison possible in the first place, which is the precondition for any serious governance policy over a portfolio of agents.
Key takeaways
- AWS shipped AgentCore Evaluations on August 26, 2026: a framework-agnostic AI agent evaluation service.
- It works with LangGraph, the OpenAI Agents SDK, LlamaIndex, Google ADK, the Claude Agent SDK, and Strands Agents, via OpenTelemetry traces (GenAI or OpenInference conventions).
- Three built-in evaluators (GoalSuccessRate, Correctness, Helpfulness), plus custom LLM-as-a-judge evaluators.
- Two modes: on-demand for CI/CD with ground truth, online for real-time production monitoring.
- The main value: comparing and supervising a mixed fleet of agents built on different frameworks without rebuilding tooling for each one.
Industrialising AI agents? SeedVision offers 3-5 day AI audits and 15-30 day production rollouts. See the packages or book a 30-min call.
Cover photo: Photo by İsmail Enes Ayhan on Unsplash.