Patterns for a sanctioned AI environment: gateway, guardrails and credentials
The usual sequence is that one team ships an AI feature with a provider key in an environment variable, it works, and within two quarters five more have done the same thing five different ways. Nothing is wrong yet. But nobody can now say which models are in use, what they can reach, whose data went to them, or how to turn one off — and each of those answers gets more expensive every month.
Published 22 September 2026. Written by the SecHB practice, Greater Vancouver, British Columbia.
The short version: the decisions that are expensive to change later
Most of a secure AI environment can be added incrementally. A few things cannot, because they are assumptions every feature is built on top of. Those are worth getting right while there is still one feature.
| Decision | The position that ages well | Why it is expensive later |
|---|---|---|
| Where credentials live | In a broker the application calls, never in client code and never in a prompt. | Moving a credential after a dozen features hold it means touching every one of them, and you cannot rotate what you cannot enumerate. |
| Whose identity a call carries | The end user’s, propagated to the broker and to every tool the request triggers. | Retrofitting per-user identity onto a system built on one service principal is a rewrite of the authorization model, not a configuration change. |
| What the model may reach | An explicit egress allowlist for model and tool traffic. | Default-open egress is trivial to add and politically hard to remove once teams depend on undocumented destinations. |
| What is recorded | Prompt, retrievals, tool calls, output and identity, correlated per session. | Telemetry cannot be added retroactively to an event that already happened. |
| How output is consumed | Treated as untrusted at every downstream sink. | Once one consumer renders model output as markup or passes it to a shell, the same assumption spreads by copy-paste. |
The rest of this page is what each of those looks like in practice.
Where the broker sits, and what it is for
A broker — gateway, proxy, internal service, the name does not matter — is a service your applications call instead of calling a model provider directly. It is worth having for four reasons, and filtering is not the main one.
- Credentials. Provider keys live in one place, are rotated in one place, and never reach an application, a client or a prompt.
- Visibility. One place records what was sent, what came back, and who asked — the substrate for both monitoring and any later reconstruction.
- Control. One place to disable a model, a feature or a tenant, and one place where limits are enforced.
- Portability. Changing provider or model version becomes a change in one service rather than in every team’s code.
The broker is also where request and token limits belong, which is the control the OWASP list treats as LLM10:2025 Unbounded Consumption — the failure mode where a model endpoint with no ceiling becomes either a denial-of-service target or an unbounded bill. Per-caller and per-tenant limits at the broker are the cheapest version of that control.
Authenticating the user, not just the application
The common shortcut is that the application authenticates to the broker with a service credential and the end user is, at most, a string in a log field. Everything downstream then runs with the application’s authority, which is invariably broader than any individual user’s.
The consequence shows up later as an authorization question nobody can answer: when the assistant retrieved that document or called that tool, was it acting within what this user could do anyway, or beyond it? If the identity was not carried, the honest answer is that you cannot tell, which turns a contained event into an open-ended one. Propagate the user’s identity to the broker, and from the broker to every tool the request triggers.
Input guardrails: what they catch and what they will not
Input guardrails inspect what is about to reach the model and block or flag some of it. They do real work: they catch careless pasting of secrets and personal information, they stop naive attempts, they generate signal, and they give a policy something to stand on.
They do not solve LLM01:2025 Prompt Injection. No. A guardrail is a classifier reading text, and an instruction it is meant to catch can be rephrased, split across sources, encoded or hidden in a format it handles poorly. Input filtering is a mitigation that lowers volume; it is not a boundary. The boundary is what the tools are permitted to do and under whose identity they do it.
That is not an argument against deploying them. It is an argument against counting them as the control that lets a model call a payments tool. Guardrails reduce noise and buy signal; permissions decide consequence.
Output guardrails and the downstream handling problem
Output guardrails check what the model produced before it goes anywhere. The more important half of this is not the classifier but the assumption underneath it: model output is untrusted input to whatever consumes it next.
The OWASP list calls this LLM05:2025 Improper Output Handling, and it is where the classic web and systems vulnerabilities come back. Output rendered as markup without escaping is cross-site scripting with an unusual author. Output interpolated into a query is injection. Output passed to a shell or an interpreter is remote code execution. Output written to a log that a console renders is a payload waiting for an operator. Escape and validate at every sink, exactly as you would for input from a stranger, because that is what it is.
Secrets, egress and per-agent credentials
Because anything in a prompt can end up in an output, in a log, in a vector store or in a provider’s context window, and anything in client code is already published. Model and tool credentials belong to the broker, which holds them server-side and exchanges the caller’s identity for scoped access per request. The Canadian Centre for Cyber Security makes the general version of the point in Generative artificial intelligence (ITSAP.00.041), dated December 2025: Do not share private information with AI tools unless you understand what they are doing with your data.
A broker is how an organization makes that instruction structural instead of advisory.
Egress. Model and tool traffic should leave through an explicit allowlist of destinations. This is ordinary network hygiene, and it matters more than usual here because the content of that traffic is your data and because a tool-calling system can be induced to fetch a destination nobody intended. An allowlist turns that from an exfiltration path into a blocked request and a log line.
Per-agent, per-user credentials. Where a model can call tools, each tool gets its own credential, scoped to that tool’s job, and the call is authorized against the user on whose behalf it runs. One broadly privileged credential shared across an agent’s tools is what turns a single successful injection into access to everything the agent could ever reach — LLM06:2025 Excessive Agency in the same list. Scope narrowly, prefer read over write, and require a human confirmation step for actions that move money, change access or contact a customer.
Logging designed for investigation, not for debugging
Debug logging answers “why did this break”. Investigation needs a different record: the prompt as assembled, the documents or tool responses retrieved and their sources, every tool call with arguments and result, the output returned, and the identity each step ran under — correlated by a session identifier so the sequence can be replayed as one trajectory.
The broker is the natural place to capture most of it. Two constraints come with it: the store now holds whatever your prompts and documents hold, so it needs retention limits and access control to match, and records nobody can query are a cost rather than a control.
Private and self-hosted deployment
Self-hosting answers questions about data residency, provider retention and model version stability, and it replaces them with the work of running inference infrastructure — patching, capacity, isolation and access control that is now yours. Decide it on the residency and contractual questions, not on a belief that local inference is inherently safer.
The questions that actually drive the decision are contractual and jurisdictional: where inference happens, what the provider retains and for how long, whether your content trains anything, whether a model version can change under you, and what your own customer commitments already say. Those are answerable from a contract and a data map. Whether any of it creates a legal obligation for a particular organization is a question for counsel.
What this does not solve, and how it maps to the frameworks
An environment built this way still does not tell you whether the retrieval index respects the permissions of its source systems, whether two tenants share something they should not, or whether the feature should exist. Those are design questions, and a gateway is not a substitute for a threat model.
For the governance wrapper, NIST AI RMF 1.0, released on 26 January 2023, and the Generative AI Profile (NIST AI 600-1), released on 26 July 2024, are the reference — and the same NIST page states that AI RMF 1.0 is being revised, which is why the version number belongs in the name whenever you cite it. For the build side, NIST SP 800-218A, Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile (July 2024), extends the Secure Software Development Framework to AI model development. Neither is a control catalogue you configure a gateway from; both are what you map the resulting environment against when somebody asks how it was decided.
Often yes, because the value is not the filtering — it is having one place where model credentials live, one place where traffic is recorded, and one switch to turn a model off. A small team can reach that with a thin internal library rather than a product, but the three properties are what matter and they are far cheaper to establish before a second and third feature are wired directly to a provider.
How the work is bounded
The scope is agreed in writing before work starts, and the engagement is quoted in writing with it.
SecHB does not issue certifications, attestations or audit opinions: those come from accredited certification bodies, CPA firms and QSAs. The work here is what an organization does to be ready for them.
Nothing here is legal advice. Where a question turns on the law, the work is done alongside the client’s counsel, not instead of them.
Questions we are asked
Is a gateway worth it for a small team?
Often yes, because the value is not the filtering — it is having one place where model credentials live, one place where traffic is recorded, and one switch to turn a model off. A small team can reach that with a thin internal library rather than a product, but the three properties are what matter and they are far cheaper to establish before a second and third feature are wired directly to a provider.
Do guardrails stop prompt injection?
No. A guardrail is a classifier reading text, and an instruction it is meant to catch can be rephrased, split across sources, encoded or hidden in a format it handles poorly. Input filtering is a mitigation that lowers volume; it is not a boundary. The boundary is what the tools are permitted to do and under whose identity they do it.
Should we self-host?
Self-hosting answers questions about data residency, provider retention and model version stability, and it replaces them with the work of running inference infrastructure — patching, capacity, isolation and access control that is now yours. Decide it on the residency and contractual questions, not on a belief that local inference is inherently safer.
Why can model credentials never sit in a prompt?
Because anything in a prompt can end up in an output, in a log, in a vector store or in a provider’s context window, and anything in client code is already published. Model and tool credentials belong to the broker, which holds them server-side and exchanges the caller’s identity for scoped access per request.
Where to go from here
Building this out is a secure AI environment, and the records it produces feed AI security monitoring. Where the system calls tools or acts on its own initiative, agentic AI security is the permission and identity design that decides blast radius, and the retrieval question this page defers is RAG tenant isolation. The practice as a whole is AI security, and the other pieces are indexed under Writing.
Describe what is deployed today and how the keys are held, and the reply will say which of these decisions is the expensive one for you.