AI threat modelling
A design review is scheduled for an AI feature that does not exist yet. Nobody in the room is sure which questions are the security questions, because the component in the middle does not behave like the rest of the stack: it cannot reliably tell an instruction from data, and it has been given credentials.
The cheapest AI engagement to buy
Threat modelling is the least expensive piece of AI security work an organization can commission, and the one that most improves everything bought afterwards. A finding removed at design costs a conversation. The same finding removed after launch costs a rebuild, a migration of a retrieval index, or a renegotiated contract with a model provider.
Threat modelling asks what could go wrong with the design; an assessment reports what is actually wrong with the build. The first is cheaper, earlier, and decides how much the second has to look at. Teams that model first almost always buy a narrower, sharper AI security assessment afterwards, because they already know which parts of the system carry the risk.
When to do it
A working system is not required, and waiting for one is the expensive choice. Threat modelling reads the design: the components, what flows between them, and where trust changes hands. That makes the design review the right moment — while the retrieval boundary is still a line on a whiteboard and the tool credentials are still a proposal.
It is also worth doing late, in two situations: when an existing system is about to be given tools or autonomy it did not have before, and when a system built for one tenant is about to serve several.
How the session runs
One or two facilitated workshops with your architects and engineers, each a half day, with a short written pre-read so the time in the room is spent on the hard parts rather than on orientation.
- Decompose. Draw the system as it is intended to work, component by component, until every flow has a source and a destination.
- Mark the boundaries. Every point where trust changes — user to application, application to model, model to tool, tool to backend, tenant to tenant.
- Enumerate. Walk each boundary against the method below and write down what crosses it and what could.
- Decide. Rank the threats, choose the controls, and record the ones consciously accepted — that record is as valuable as the mitigations.
What we decompose
- Models and weights. Hosted or self-run, pinned or floating, fine-tuned on what, and who can change any of it.
- Prompts. The system prompt, the templates, and every place content the organization does not control is concatenated into one.
- Retrieval corpus. What is indexed, who could get a document into it, and whether the index carries the permissions of the source system.
- Tool credentials. What each tool can do, whose authority it borrows, and whether that authority exceeds the requesting user’s.
- Identities. End user, service account, agent — and the hops at which one silently becomes another.
- Trust boundaries. The lines the rest of the model is organized around, including the one most designs omit: model output re-entering the application as data.
The six boundaries, and the question asked at each
Most of the value comes from asking one specific question at each crossing rather than a general one about the system. These six recur in nearly every AI design, and a model that answers all of them is already more use than most security reviews produce.
| Trust boundary | The question |
|---|---|
| User to application | Who is asking, what are they entitled to, and does the model inherit their authority or something wider? |
| Untrusted content to model | Which channels bring in text the organization does not control, and is any of it delimited and marked as data rather than instruction? |
| Model to tool | What can be invoked, with whose credentials, and which invocations are irreversible? |
| Tool to backend | Does the backend re-check authorization, or does it trust the caller because the caller is internal? |
| Tenant to tenant | What enforces separation in the retrieval index and in memory — the store itself, or application code that can be bypassed? |
| Model output to application | What renders, executes, queries or stores the output without treating it as untrusted input? |
The last row is the one most designs omit, because model output feels like a result rather than like input arriving from an untrusted source. It is both.
Method: STRIDE, applied to AI-specific assets
The structure is ordinary threat modelling. STRIDE — spoofing, tampering, repudiation, information disclosure, denial of service and elevation of privilege — is walked against each boundary, because it is the vocabulary your architects already share and because most AI failures are recognizable members of those six families once the asset is named correctly. Tampering with a retrieval corpus is still tampering. An agent borrowing a service account is still elevation of privilege.
What AI changes is the asset list, so the threat checklist is the OWASP Top 10 for LLM Applications 2025: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. Each is asked at the boundary where it would arrive.
For teams that want an adversary-behaviour frame alongside the design view, MITRE ATLAS catalogues real-world attacks on AI-enabled systems and is named here at framework level. Governance-side threats — an unowned model, an unrecorded prompt change, no decision rights for approving a system — map to NIST AI RMF 1.0, so they land on a named owner rather than being written up as engineering work nobody schedules.
What you receive
- A system decomposition — the diagram your design review should have had, with every trust boundary drawn.
- Enumerated threats, each mapped to the boundary it crosses and the OWASP category it belongs to, so nothing is a free-floating worry.
- Prioritized controls, separated into what must exist before launch, what can follow, and what is being accepted deliberately.
- A reusable template your team maintains, so the second system is modelled by your own architects in an afternoon.
The model belongs to your team, not to us: it ships as a template your architects update when the system changes, and the threats it enumerates become tickets, design constraints and acceptance criteria. For the shape of the artifact, see what a threat model looks like; for the controls the model usually asks for next, see secure AI environment establishment.
How the work is bounded
The scope is agreed in writing before work starts, and the engagement is quoted in writing with it.
SecHB does not issue certifications, attestations or audit opinions: those come from accredited certification bodies, CPA firms and QSAs. The work here is what an organization does to be ready for them.
Nothing here is legal advice. Where a question turns on the law, the work is done alongside the client’s counsel, not instead of them.
Questions we are asked
Do we need a working system first?
A working system is not required, and waiting for one is the expensive choice. Threat modelling reads the design: the components, what flows between them, and where trust changes hands.
Who needs to be in the room?
The people who can answer "what happens if this input is hostile" — the architect, the engineers who built the retrieval and the tool integrations, and whoever owns the data the system reaches. Four to eight people is the workable range. A product owner who can say what the feature must do is more useful than a second security specialist.
How is this different from an AI security assessment?
Threat modelling asks what could go wrong with the design; an assessment reports what is actually wrong with the build. The first is cheaper, earlier, and decides how much the second has to look at.
What do we do with the threat model afterwards?
The model belongs to your team, not to us: it ships as a template your architects update when the system changes, and the threats it enumerates become tickets, design constraints and acceptance criteria. A model that is never opened again was a report; a model your team edits at the next design review is a control.
Book a design review
Send the architecture as it stands — a diagram, a design document or a description. The reply says what the workshops would cover and who should be in them.