MLOps

LiteLLM on Bedrock: The Gateway That Budgets Your AI Agents

AWS documents a self-hosted LiteLLM gateway on ECS and Bedrock to budget and log AI coding agents like Codex — architecture, limits, and an IAM alternative.

LiteLLM on Bedrock: The Gateway That Budgets Your AI Agents

LiteLLM on Bedrock: The Gateway That Puts a Budget on Your AI Agents

A developer wires up OpenAI Codex to your AWS account. Three weeks later, someone in finance asks why the Bedrock bill tripled. On September 3, 2026, AWS published a full deployment walkthrough for exactly this problem: route Codex through a self-hosted LiteLLM gateway on Amazon ECS, sitting in front of Amazon Bedrock. It's not a configuration trick — it's a complete architectural pattern, with its operational cost stated up front rather than glossed over.

The background

An LLM gateway sits between your developers' tools and the AI models they call. Instead of every developer holding their own API key to a model provider, all traffic routes through a single control point that can enforce quotas, log consumption, and hide the real credentials behind a service identity. LiteLLM is the open-source project playing that role here: a unified API in front of 140+ model providers, built with cost controls and governance for platform teams in mind. AWS's guide documents how to self-host it on ECS to govern access to Amazon Bedrock — specifically for OpenAI Codex used as a coding agent.

A LiteLLM architecture defined entirely as infrastructure-as-code

The reference deployment runs entirely on managed AWS building blocks, orchestrated through CloudFormation:

  • Amazon ECS on Fargate hosts the LiteLLM gateway containers
  • Application Load Balancer + AWS WAF filter and control inbound traffic
  • Amazon RDS (PostgreSQL) stores LiteLLM state, usage metrics, and budget data
  • AWS Secrets Manager + AWS KMS handle credentials and key encryption
  • Amazon ECR holds an immutable gateway container image
  • Amazon CloudWatch centralizes logs, alarms, and Container Insights

The end-to-end flow runs in five steps: Codex sends task context and tool definitions to the gateway's /v1/responses endpoint; the ALB and WAF apply their controls before routing the request to LiteLLM on Fargate; LiteLLM authenticates the caller, validates the attached policy, then invokes the Bedrock model through the ECS task's IAM role; Bedrock returns text or function calls back through LiteLLM; Codex executes the approved tools locally and returns the result on the next request.

Immutable images, automatic rollback

The deployment uses a digest-pinned Docker image, pushed to ECR and passed to CloudFormation by digest — never a mutable tag. The ECS service ships with a deployment circuit-breaker that triggers an automatic rollback on failure, backed by ALB health checks. Target-tracking autoscaling flexes the fleet between 2 and 10 tasks in the reference setup, and RDS can run multi-AZ with automated backups.

An alias, never the provider's real identifier

Alias-to-model mappings live in a litellm_config.yaml file, rebuilt into the container image on every deploy — for instance, the alias gpt-5.5 resolves internally to bedrock_mantle/openai.gpt-5.5. Developers and Codex only ever see the stable alias: the gateway absorbs any change of provider, region, or model version without touching a single workstation. That decoupling is what makes the pattern portable beyond Bedrock — the same LiteLLM instance can, in principle, route other aliases to entirely different model providers without changing a line of calling code.

LiteLLM quotas scoped per key, not per developer's good faith

The core of the pattern is identity attribution: every team or developer gets their own gateway API key — never the master credential — provisioned through LiteLLM's /key/generate API and stored in a KMS-encrypted Secrets Manager secret. On the Codex side, the token is fetched at runtime by a Python helper script that calls the AWS CLI, and it never sits in a plaintext config file.

Each key carries concrete consumption policies: a maximum budget (the walkthrough's example sets $50 over 30 days), a tokens-per-minute limit (100,000 in the example), a requests-per-minute limit (1,000), and a restricted list of approved models (a single alias in the example given). Once a threshold is crossed, LiteLLM simply rejects the next request — no after-the-fact alert, no surprise invoice at month-end. None of this touches the model provider directly: Bedrock never sees a per-developer identity, only the shared ECS task role, which is precisely what keeps the enforcement point close to the caller instead of buried in a monthly billing export nobody reads until it's too late.

Telemetry built for accountability, not just debugging

LiteLLM's request-log view records, per call: success or failure status, the key alias (preserving developer identity, while Bedrock itself only ever sees the ECS task's IAM role), the model that actually got resolved, token counts, request duration, and time to first token. A tool-using task produces multiple log rows, which makes the execution boundary explicit: LiteLLM governs and logs the model calls; Codex runs the tools locally, on the developer's own machine.

Why coding agents make this urgent

Chat assistants burn tokens at the pace of a human typing questions. A coding agent like Codex doesn't: once it's given a task, it can loop through dozens of tool calls, file reads, and follow-up completions with no human in between, each one a billable Bedrock invocation. That's exactly the gap a per-request budget closes and a monthly invoice review can't — by the time a finance dashboard shows the spike, the agent has already run for weeks. A hard per-key ceiling, enforced on the very next call rather than reported after the fact, is what turns "we'll notice eventually" into "it simply can't happen."

What this means for AI teams

This pattern sits in direct contrast to another Bedrock spend-governance approach we covered recently on this blog: Jamf's, which drives spend quotas straight off IAM policy, with no extra infrastructure to run at all. Both patterns solve the same underlying problem — an unchecked Bedrock bill — with opposite trade-offs.

The pure-IAM pattern (Jamf) needs no running service: the restriction lives in the cloud's native authorization layer. But it offers no per-request granularity, no detailed conversation log, and stays tied specifically to Bedrock. The gateway pattern (LiteLLM on ECS) flips that: per-key, per-request granularity, portability across model providers, richer observability — at the cost of infrastructure you now have to keep alive. AWS's guide is upfront about that trade-off: "your team owns gateway availability, database lifecycle, version upgrades, incident response, and capacity planning." That's not a footnote — it's a new production system, with its own on-call rotation.

Three takeaways carry over directly to a team operating AI agents in production:

  • The right pattern depends on team maturity. A small team with a simple need is better served starting with direct access through IAM Identity Center — fewer moving parts, native traceability via AWS CloudTrail. A dedicated gateway earns its keep once per-team granularity, multi-provider routing, or fine-grained observability become real needs, not hypothetical ones.
  • Quotas don't substitute for content filtering. The guide is explicit: LiteLLM's controls (budgets, rate limits) aren't sufficient alone in production — Amazon Bedrock Guardrails needs to sit on the model-access path for content filtering, denied-topic detection, and grounding checks. Governing spend and governing content are two separate problems.
  • A contract-validation script is not a load test. The guide is blunt about this: before production, also test concurrent agent sessions, long-running streams, request cancellation, key revocation, and failure recovery — well beyond simple /v1/responses compatibility.

The guide also flags a managed alternative, Portkey, for teams that want gateway benefits without operating the infrastructure themselves — at the cost of tighter control and a vendor dependency worth weighing case by case.

Two more operational details, easy to miss but just as binding, are worth planning for before reproducing this pattern. First, model availability varies by AWS account and region — the reference deployment was only validated on us-east-1, so availability needs re-checking before deploying anywhere else. Second, whether to log prompts and responses at all isn't a default decision: the guide explicitly frames this content as potentially sensitive customer data, requiring its own redaction, encryption, access-control, and retention policy — including after the CloudFormation stack is deleted, since RDS snapshots, KMS keys, secrets, and CloudWatch logs are retained by default according to your data-retention policy.

Key takeaways

  • AWS documents a self-hosted LiteLLM deployment on ECS Fargate in front of Amazon Bedrock to govern OpenAI Codex access, published September 3, 2026.
  • Every developer gets a dedicated gateway key with its own budget, rate limits, and approved-model list — the next request is rejected the moment a threshold is crossed.
  • Per-request telemetry (status, tokens, duration, time to first token) preserves developer identity, while Bedrock only ever sees the ECS task's IAM role.
  • This gateway pattern is the opposite trade-off from a pure-IAM approach (like Jamf's): more granularity and portability, at the cost of infrastructure you now operate.
  • The guide is clear that quotas don't replace content filtering (Bedrock Guardrails), and that a compatibility script isn't a load test.
  • A managed alternative (Portkey) exists for teams that want the benefits without the operational burden.

Industrialising AI agents? SeedVision offers 3-5 day AI audits and 15-30 day production rollouts. See the packages or book a 30-min call.

Cover photo: Photo by Growtika on Unsplash.