MLOps

Cutting AI Agent Costs in CI: GitHub's 5-Lever Method

GitHub published its internal method for cutting agentic CI token consumption — up to 62% savings on some workflows. Five concrete levers to apply.

Cutting AI Agent Costs in CI: GitHub's 5-Lever Method

Cutting AI Agent Costs in CI: GitHub's 5-Lever Method

GitHub published an unusual postmortem on May 7, 2026: Landon Cox and Mara Kiefer, Microsoft Research authors, walk through their method for cutting token consumption on their own CI agents. The numbers are concrete: −62% on Auto-Triage Issues, −79% on Smoke Claude when stacking multiple levers.

For any team that just discovered through Copilot's move to usage-based billing that AI agent costs in CI aren't invisible anymore, this is the practical playbook to apply. We unpack the five levers GitHub uses internally — applicable to any agentic pipeline, starting next week.

The context: why CI agents drift over time

GitHub Agentic Workflows — the automated pipelines that run on every pull request for triage, docs, quality checks, security review — have one defining trait: they execute the same work every time, unlike an interactive session. That regularity makes them predictable, which makes them instrumentable, which makes them optimisable.

But that same regularity also drives the bill up silently. A workflow running 7 times per day at 50,000 tokens per run is 10.5 million tokens per month for one workflow. Multiply by 8 agents in CI and you're at 84M tokens/month — that's $84 on Claude Sonnet, $420 on Opus, and nobody sees it in the cost dashboard because it's buried in the seat-based plan.

The shift to usage-based pricing affecting Copilot, the Claude API, and the OpenAI API changes the relationship to noise: now every wasted token has a readable marginal cost.

The Effective Tokens metric: comparing apples to apples

Before optimising, you measure correctly. GitHub defines a unified metric called Effective Tokens (ET) that weights tokens by their actual cost:

ET = m × (1.0 × I + 0.1 × C + 4.0 × O)

Where m is the model multiplier (Haiku 0.25, Sonnet 1.0, Opus 5.0), I is fresh input tokens, C is cache-read tokens, O is output tokens. Why these weights? Output tokens are typically 4× more expensive than inputs across all major providers, and cache reads are billed at roughly 10% of a fresh input.

The point of ET is that it lets you compare a workflow before/after a model switch, workflow A vs B, or a single workflow's evolution over time. Without that common denominator, you compare apples and oranges.

The five concrete levers

Lever 1 — Log before you optimise

GitHub recommends an API proxy that captures every model call into a normalised token-usage.jsonl file. One line per call with input tokens, output tokens, cache-read, cache-write, model, timestamp. The instrumentation is independent of agent framework (Claude CLI, Copilot CLI, Codex CLI) and gives you the historical view any decision needs.

Cox and Kiefer are categorical: "Add the API proxy, turn on logging, and let the data tell you where to look." Without that foundation, you optimise blind. With it, you identify the top 3 token-hungry workflows and concentrate effort where it pays back.

Tooling: gh extensions install github/gh-aw then gh aw add githubnext/agentic-ops/copilot-token-audit copilot-token-optimizer adds two automated workflows — a daily auditor that aggregates consumption and an optimiser that opens GitHub Issues with concrete optimisation proposals.

Lever 2 — Prune unused MCP tools

MCP (Model Context Protocol) standardises tool declarations available to an agent. But every declared tool sends its JSON schema in the context of every call. The default GitHub MCP server provides 40 tools; that translates to 10 to 15 KB of schema sent on every turn, even if the agent only uses 2 of those 40.

The audit is mechanical: cross-reference the declared tool manifest against the tool calls actually made on recent runs. Disable everything that's never called. GitHub reports the pruning drops per-call context by 8 to 12 KB — several thousand tokens per run with no behavior change.

Extreme case documented: on the Glossary Maintainer workflow, a single tool (search_repositories) had been called 342 times in one run, accounting for 58% of all tool calls. Spotting that hot-spot let the team rework the agent strategy and bring consumption down dramatically.

Lever 3 — Replace MCP with GitHub CLI when you can

An MCP tool call is a full LLM round-trip: the agent decides to call the tool, formulates arguments, receives the response, interprets it. Three token-billed steps. By contrast, gh pr diff is a deterministic HTTP request with no LLM in the loop.

Two concrete patterns:

Pre-agentic data download: before the agent starts, GitHub Actions steps run the necessary gh commands and write results to workspace files. The agent reads those as context instead of fetching them via MCP.

In-agent CLI proxy: a lightweight HTTP proxy redirects CLI calls to the GitHub API without exposing auth tokens to the agent. The agent sees a local endpoint.

The authors' mantra: "The cheapest LLM call is the one you don't make." For any flow where the data to retrieve is deterministic, take it out of the agentic loop.

Lever 4 — Pick the model per task

Not every workflow needs Claude Opus 4.7. An agent formatting a log message can run on Haiku, which costs roughly 4× less per token than Sonnet. The quality profile is still plenty for "transform A to B" tasks.

The documented winning combo: Smoke Claude in the gh-aw-firewall repo dropped consumption by 79% by stacking aggressive MCP pruning with a switch to Haiku. The quality trade-off was monitored via process signals (output tokens per call, tool-call completion rates), not via direct correctness measurement — which raises a methodology question covered below.

Lever 5 — Relevance gates

The most radical lever: skip the LLM entirely when the use-case condition isn't met. The Security Guard workflow now only launches its agent on PRs that modify files in the security perimeter; for everything else, it exits in milliseconds without spending a token. Result: −43% ET on aggregate.

This logic generalises. Every agentic workflow should have a conditional gate at the top that answers "does this run have any chance of being useful?". When the answer is "probably not", you skip.

The trap: measuring real efficiency

Three confounding factors prevent a naive reading of gains:

Different models = different per-token cost. Without the ET metric, model switch and token optimisation are indistinguishable.

Workload variability. The Contribution Check workflow showed a 5% ET increase after optimisation — not because the optimisation failed, but because the post-fix period coincided with a shift toward larger PRs (65% large vs 41% before). The optimisation worked but was masked by context.

No ground-truth quality signal. Nobody can tell if an agent "did the right work" on open-ended tasks. Process signals (output tokens, turn count, tool-call completion rate) approximate quality but don't measure it. Cox and Kiefer concede: "Measuring tokens-per-unit-of-correct-work requires additional instrumentation and thought."

What this actually changes for AI teams

Three concrete implications for anyone running agents in production:

1. Agentic observability is non-negotiable. Before any optimisation effort, instrumentation. Without token-usage.jsonl or equivalent, you don't know where to look. It's the same logic as the AI Act event-logging requirement — the technical base is shared, the use case differs.

2. Portfolio beats unit. Run frequency × per-run savings is the metric that matters. Auto-Triage Issues at 6.8 runs/day with −62% generated the highest dollar saving in GitHub's portfolio (~7.8M ET over the observation window). Optimising a rare high-volume-per-run workflow returns less than a frequent moderate-volume workflow.

3. LLM optimisation suggestions need to be instrumented, not blindly applied. The daily optimiser GitHub deploys opens Issues with proposals; not all of them translate to measurable gains. On short windows, workload noise masks improvements. Practical stance: apply the optimisations, but watch ET over 4-6 weeks before concluding.

For a team that just wired 3 or 4 agents into CI, the order of operations is clear: add the logging proxy (D), run an MCP audit (D+5), introduce relevance gates (D+10), experiment with model switches on non-critical workflows (D+15). Three weeks to target a −40% structural drop without touching quality is realistic.

TL;DR

  • GitHub publishes May 7, 2026 an internal postmortem on token optimisation for CI agents: −62% on Auto-Triage, up to −79% stacked
  • Effective Tokens metric: ET = m × (1.0×I + 0.1×C + 4.0×O) to compare across models
  • 5 levers: 1) log before you optimise via API proxy, 2) prune unused MCP tools, 3) replace MCP with GitHub CLI when calls are deterministic, 4) switch model per task (Haiku ~4× cheaper than Sonnet), 5) relevance gates to skip the LLM
  • Traps: workload shift, no ground-truth quality signal, models not comparable without ET
  • Practical order: observability (D) → MCP audit (D+5) → relevance gates (D+10) → model switch (D+15). Realistic target: −40% structural in 3 weeks.

Industrialising AI agents? SeedVision offers 3-5 day AI audits and 15-30 day production rollouts. See the packages or book a 30-min call.

Cover photo: Photo by 1981 Digital on Unsplash.