One 'AI Gateway', Two Different Jobs: the LLM Gateway and the Tool Gateway

Every agentic platform conversation I have eventually collides with the phrase “AI gateway”. And I have lost count of how often the two people saying it are talking about two different products without noticing.

One of them means a proxy in front of models — the thing applications call instead of calling OpenAI, Anthropic, or Amazon Bedrock directly. The other means a front door for tools — the thing agents call instead of hitting internal APIs directly. Both get called “the gateway”. They sit on different traffic, solve different problems, are bought or built on different criteria, and a verdict about one says nothing about the other.

The confusion is not cosmetic. I have watched it produce migration plans that could not work, and gateway components justified by criteria that collapse under one documentation link. This post is the untangling I now do reflexively, written down.

The two roles

An LLM gateway proxies model traffic. It sits between your applications (or your agents) and the model providers. The integration contract is famously thin: applications change a base URL, a credential, and a model name. Everything the gateway does lives on that path — provider abstraction, model routing and fallback, credential custody (application teams never hold provider keys), per-team or per-tenant metering and quotas, and centralized logging of prompts and completions. LiteLLM and TrueFoundry are archetypes of the category; on AWS, Amazon Bedrock itself plays much of this role natively, since it already fronts many providers behind one endpoint, one IAM credential model, and one metering plane.

A tool gateway fronts tool traffic. It sits between agents and the capabilities they invoke — internal APIs, SaaS, databases, other agents. In 2026 that increasingly means it speaks MCP to the agent and whatever the backend speaks on the other side. Its concerns have almost nothing in common with the model path: tool admission (which tools exist at all, and who approved them), per-agent authorization (which agent may call which tool with which scopes), schema and version control of tool contracts, and audit of every invocation — because tool calls are where agents touch systems that have side effects.

The reason the vocabulary collapses: one product can play both roles. Amazon Bedrock AgentCore Gateway is the clean example — its inference targets proxy model traffic (an LLM-gateway role), while its MCP targets front tools (a tool-gateway role). Same service name, two unrelated jobs. If you say “we use AgentCore Gateway” in an architecture review, you have said almost nothing yet; the meaningful sentence names the role: “AgentCore Gateway, inference targets, as our LLM gateway” or “AgentCore Gateway, MCP targets, as our tool gateway”.

So the first habit is cheap and mechanical: every time the word “gateway” appears — in a diagram, a vendor deck, a migration plan — annotate it with the role. LLM gateway (model traffic) or tool gateway (tool traffic). I have yet to sit through a discussion where forcing that annotation did not surface at least one hidden disagreement.

Trap one: a verdict on one path applied to the other

“We are replacing the gateway” is an incomplete sentence. Replacing your LLM gateway changes how applications reach models — base URLs, credentials, format translation, metering. It does not move a single tool. Replacing your tool gateway changes how agents reach capabilities — admission, authorization, audit. It does not affect which models you can call or how.

The failure mode is a go/no-go decision made on one path and silently inherited by the other. A team evaluates a platform’s model-routing capabilities, finds them sufficient, and declares “we can drop the gateway” — while their agents’ MCP admission and audit story quietly evaporates, because that job was done by the same box under the same name. Or the inverse: a tool gateway is mandated for governance reasons, and someone concludes that all model traffic must now hairpin through it too, adding a hop that re-creates nothing the platform did not already provide.

Scope every verdict to the role it was evaluated against. Two decisions, two criteria sets, even when one product ends up doing both.

Trap two: assuming the gateway translates

Here is the one that breaks migrations, because it hides inside the thin integration contract.

The promise of an LLM gateway is “change the base URL and the credential, keep your code”. That promise silently assumes the API format on both sides matches — that your OpenAI-SDK application can speak to whatever sits behind the gateway. Whether that is true depends on which kind of gateway you have:

• A translating gateway (the LiteLLM/TrueFoundry class) accepts one format — typically the OpenAI Chat Completions dialect — and rewrites requests into each provider’s native API. Your code genuinely does not change.
• A pass-through gateway (AgentCore Gateway inference targets, and most “unified endpoint” designs) routes and governs the traffic but forwards the request body as-is. The application must already speak the target model’s native format.

And “does the target speak OpenAI?” is a per-model property, not a per-endpoint property. This is the detail I now verify line by line, because it is counterintuitive: on Amazon Bedrock, as of late September 2026, the OpenAI-compatible Chat Completions and Responses endpoints exist for GPT-OSS and GPT-5.x models, Mistral Large 3, Qwen3, and DeepSeek V3.x — and for no Claude, no Nova, and no Llama model. Recent Claude models on Bedrock speak the Anthropic Messages API (or Converse); embedding models speak Invoke only. The model API compatibility page is the source of truth, and it is a table with one row per model, not a banner over the endpoint.

So the sentence “our applications use the OpenAI library, we will just point them at the platform” is not a plan; it is a hypothesis that is true or false per target model. If the models you actually want are in the incompatible column, you have exactly two honest options: put a translating gateway in the path, or change the application’s SDK. A pass-through hop — however governed, metered, and well-logged — does not manufacture compatibility.

Trap three: justifying the gateway by vendor reach

The weakest justification I see for adding a dedicated LLM gateway on top of a platform is some variant of “we need it for the non-native providers”. It sounds reasonable and dies on contact with documentation, because platform reach keeps expanding — model catalogs grow, and agent runtimes are increasingly model-agnostic (Bedrock AgentCore Runtime, for instance, is documented to work with models inside or outside Bedrock). A justification anchored on vendor reach has a shelf life of one product announcement.

The durable criterion is different: an extra gateway earns its place by the controls it re-creates for traffic your platform’s native controls cannot see. Concretely, name the traffic and name the controls:

• Traffic: calls to providers outside the platform’s catalog, or from consumers outside the governed runtime — the flows that would otherwise leave your estate ungoverned.
• Controls: a single egress point for that traffic, per-team/per-tenant metering where the platform cannot meter, evidence capture and guardrail enforcement, and key custody so application teams never hold external provider credentials.

If a flow is already visible to native controls — already metered, logged, guarded, and credentialed by the platform — an extra hop on that flow needs a better reason than symmetry. If a flow is not visible, the gateway is not an “abstraction layer”; it is the control plane for that flow, and you should be able to list which controls it provides. Justifications that survive are lists of controls. Justifications that die are lists of vendors.

The checklist I actually use

When an “AI gateway” lands on the table, in any direction — adding one, removing one, replacing one — I walk these five questions:

1. Which traffic? Model calls or tool calls? Annotate the role before discussing the product.
2. Two inventories. Who calls models, and through what? Which agents call which tools, and through what? These are separate lists with separate owners.
3. Per role, which controls are needed — and what already provides them? Metering, custody, audit, admission, guardrails. “The platform already does this natively” is a legitimate and frequent answer on the model path; it is rarely the answer on the tool path once agents multiply.
4. Per target model, does the format match? Check the compatibility table row by row. Never per endpoint, never by vendor reputation.
5. Scope the verdict. Write the decision as “LLM gateway: X” and “tool gateway: Y” — even when X and Y happen to be the same product.

Honest recommendation

If you run on one cloud with one model platform, you probably do not need a separate LLM gateway on day one — the platform is your LLM gateway: one endpoint, one credential model, native metering. Add a dedicated one when a real flow appears that the native controls cannot see (external providers, consumers outside the governed runtime), and justify it by the controls it re-creates, never by the vendor list it unlocks. If your applications are standardized on the OpenAI SDK, decide with the compatibility table open: for some model families the pass-through story simply does not exist, and pretending otherwise just relocates the migration to the worst possible moment.

The tool gateway conversation deserves the opposite bias: it earns its place early in agentic systems, because tool admission, per-agent authorization, and invocation audit have no native equivalent once you run more than a handful of agents — that is where the blast radius lives.

Mostly: refuse to let one word carry two decisions. Every time “gateway” comes up, ask which one. It is the cheapest disambiguation in agentic architecture, and in my experience the single most clarifying.