What a gateway is
A gateway is a service that sits between the client tools and the model providers. Every request to a model goes through it; nothing is sent directly.
- The organisation holds the real provider credentials at the gateway. They are not distributed to individuals or stored on workstations.
- The gateway issues its own internal keys, one per user or per application. These are the only keys client tools ever hold.
- The client tool sees one base URL and one API key, as it would with a provider directly. It does not need to know which provider, or how many, are behind the gateway.
From the user’s side the change is small: a different endpoint and a different key. Everything else about the pattern happens on the gateway side.
Why organisations choose it
The reasons are mostly about control and accountability:
- Central logging and audit. One place records who sent what, when, to which model.
- Key lifecycle without sharing real keys. Internal keys can be issued, rotated, expired and revoked per person or per application. The provider credential never leaves the gateway.
- One place for sensitive-data filters. A filter that applies to every request only has to be built and maintained once.
- Cost allocation and limits. Usage can be attributed, and capped, per person or per project.
- No lock-in to one provider. Clients are written against the gateway, so the provider behind it can be changed or added to without touching every client.
- A single approved exit from an internal network. Where internal networks have no general access to the Internet, the gateway is often the only approved route out.
An organisational decision first
Adopting a gateway is, first of all, how an organisation makes the use of AI approvable: security, audit and budget owners each get something they can sign off. It is not mainly a technical optimisation, and it does not make requests faster or better. Judge it by whether it makes the use of AI acceptable to the people who must approve it.
What it controls, and what it does not
A gateway does not stop data from leaving the organisation. The model runs outside, and a request that is allowed through carries its content with it. What the gateway does is make that outflow controlled and traceable: there is a single path, it is logged, it can be filtered and limited, and access to it can be withdrawn.
A filter can only catch what it is configured to recognise. Treat it as one layer, not as a guarantee.
What is actually sent
A common misunderstanding is that the content of a request is what the user typed. With an AI coding assistant it is more than that:
- the prompt the user typed;
- the files the tool has read, in whole or in part;
- the output of commands the tool has run;
- the history of the session so far;
- context documents from the repository, such as instruction files the tool loads automatically.
A short question can therefore bring a large amount of source code with it, because the tool decides what to read in order to answer. Two habits follow from this:
- Name the files that are relevant instead of letting the tool scan the repository. See shortcuts and habits.
- Clear the context when switching to another task, so that material from the previous task is not sent again. See
/clearin CLI operations.
Trade-offs to know in advance
These follow from the design. They are costs accepted on purpose in exchange for the control above, not defects.
| Trade-off | What it means in practice |
|---|---|
| Provider features may be reduced | A layer that presents many providers through one interface may not pass every provider-specific feature through, or may pass it through in a weaker form. Prompt caching is the most important example; see cost. Which features work is a property of the specific gateway and endpoint, so ask (see the questions below). |
| The gateway decides which models exist | The set of available models is whatever the gateway exposes. A model the provider offers may not be available to you. |
| Model names may be aliases | A gateway may map a name you see to an internal alias, and the alias to whatever model is configured behind it. The model that actually answers can differ from the name displayed, and can change without any change on your side. This matters for reproducibility; see the CI checklist on pinning and recording the model. |
| One more network hop | Every request has an extra step to make, and the gateway is one more thing that can be slow, limit your rate or be unavailable. When it is down, everything behind it is down. |
Effect on cost
This pattern is often paired with usage-based billing (paying for what is consumed) instead of a fixed price per seat. Two consequences are worth understanding before the first invoice.
- Long sessions get more expensive per turn. The whole history is sent again with each request, so the cost of a turn grows with the length of the session. A long session left open is not free just because it is idle between questions. Clearing context between tasks is a cost habit as well as a data habit.
- Repeated material in automated runs. A pipeline that sends the same block of documents on every run pays for that block every time. Whether the repeated part is billed at a lower rate depends on whether prompt caching works through the gateway. For an unattended pipeline, that difference can dominate the running cost.
This page gives no prices. Rates and the rules for caching change; use the provider’s official pricing and prompt-caching documentation, and the gateway team’s statement of what applies through their endpoint. Per-run and per-project limits are covered in the enterprise questions.
Checklist: which environment am I in
Four read-only checks. Names of settings and variables differ between tools and versions, so each points to the tool’s own documentation instead of guessing.
- Endpoint and proxy. Look at the environment variables and settings that set the endpoint (base URL) and any proxy. Where they are called and where they live depends on the tool and version; check the tool’s documentation for the current names. A base URL that is not the provider’s own is the clearest sign of a gateway.
- Status. Run
/statusin the session to see the model in use and which settings sources were loaded. This tells you where the configuration comes from. See CLI operations. - Model name. Compare the name shown with the names the provider publishes. A name that does not match, or that looks like an internal label, suggests a mapping. Ask the gateway team what it resolves to.
- Deny rules. Read the list of blocked actions in the settings files (
/permissionsshows the rules in effect). They show where the organisation has drawn its boundary. See configuration worth having.
Reading is the whole job
Do not edit the configuration to remove a block, and do not probe internal systems beyond the scope you were given. A block is a decision somebody made; if it gets in your way, ask the team that owns it. Working around it can breach policy even where it is technically possible.
Questions for the gateway team
Written generally, so they apply to any gateway. Keep the answers with the rest of the adoption record; the full template is in the enterprise questions.
- Is the endpoint native to the provider’s API, or a compatibility layer?A compatibility layer is where features most often go missing or behave differently.
- Does prompt caching work through it?Ask for a yes or a no in writing, and what changes about what is stored.
- Which models are available, and are versions pinned?Pinned means a fixed version. An alias can move without notice.
- Are model names mapped to internal aliases?If yes: to which models, who changes the mapping, and how are you told.
- In which geographic region is data processed?Both at the gateway and at the provider behind it.
- Do the logs store content or only metadata? For how long, and who can read them?Content logs hold source code and need to be treated as such.
- What are the quotas and rate limits, and what happens at the limit?Per user, per application and per project, and whether a limit blocks or only reports.
- How long does a key last, and how is it renewed?Include what happens to a running pipeline when its key expires.
Unofficial notes, not affiliated with or endorsed by Anthropic. Command names, flags and behaviour change between versions; the official documentation and the tool's own help output are authoritative.