AI red teaming and AI safety evaluation answer different questions
Two proposals arrive. Both use the phrase “AI red teaming”. One describes attacking your deployed assistant until it gives up data it should not; the other describes running a battery of prompts at a model and reporting how often it refused. Both are legitimate work. Only one of them answers the question your customer asked.
Published 22 September 2026. Written by the SecHB practice, Greater Vancouver, British Columbia.
The short version
Adversarial security testing asks whether an attacker can make the system do something harmful to you. Safety evaluation asks whether the model behaves acceptably. The first is about reach and consequence; the second is about output quality against a standard of behaviour.
Three questions separate them in any proposal: what is the target, what counts as success, and what is the deliverable. Adversarial security testing targets a deployed system, succeeds by achieving an attacker objective, and delivers findings with reproduction steps; evaluation targets a model, succeeds by measuring behaviour against a rubric, and delivers scores.
What is AI red teaming?
In the security sense, it is adversarial, objective-based testing against a deployed system and everything it is wired into. The tester is not working through a list; the tester has a goal — read another tenant’s documents, make the assistant issue a refund, get the model’s tool credentials to do something the user could not do directly — and takes whatever path the system allows.
The OWASP GenAI Red Teaming Guide frames the scope in four areas: model evaluation, implementation testing, infrastructure assessment, and runtime behavior analysis
. Note that model evaluation is one of four, not the whole thing — which is exactly where the confusion in procurement comes from. The threat content usually comes from the OWASP Top 10 for LLM Applications, and the adversary behaviour vocabulary from MITRE ATLAS, which catalogues how real attacks against AI-enabled systems have been carried out.
What is safety evaluation?
Benchmark-driven measurement of model behaviour. A defined input set, a defined rubric, a score, and the ability to run the same thing again after a model or prompt change and compare. It is the discipline that tells you a fine-tune made the model more willing to produce something you did not want, or that a provider’s silent update changed how your assistant answers.
It is genuinely valuable and genuinely not security testing. It runs on expected inputs rather than hostile ones, it measures the model rather than the system, and its output is a number rather than a reproducible attack.
What each one finds
| Dimension | Adversarial security testing | Safety evaluation |
|---|---|---|
| The target | A deployed system: the application, its retrieval layer, its tools, its identities, its infrastructure. | A model, or a model plus a system prompt, usually in isolation. |
| The objective | Reach something you should not be able to reach, or make the system act in a way it should not act. | Determine whether the model’s outputs fall within an acceptable range on a defined set of inputs. |
| The method | Adversarial and objective-led. The tester chooses the path, adapts when it fails, and chains weaknesses that are individually minor. | Benchmark-led and repeatable. The same inputs are run again to compare versions. |
| Success looks like | A working attack chain, written up so an engineer can reproduce it. | A score, a distribution, or a pass rate against a rubric. |
| What it tells a buyer | What an attacker can do to you through this system today. | How the model tends to behave, and whether a change made it worse. |
| What it misses | Behaviour on the ordinary inputs nobody attacked. | Everything outside the model: retrieval scope, tool permissions, output handling, identity, infrastructure. |
No. A benchmark score describes how a model responded to a fixed set of prompts under laboratory conditions; it says nothing about whether your retrieval layer leaks across tenants, whether a tool is wired in with more permission than it needs, or whether an uploaded document can redirect the assistant.
What neither one finds
Both are exercises against something that already exists, which means both inherit whatever the design got wrong. Neither will tell you that the retrieval index was built by a service account with read access to every document in the organization — a tester might stumble into it, but only if the test account happened to sit on the right side of the gap.
Neither will tell you that the same model instance serves two customers whose contracts say it does not. Neither will tell you that no one can say which model version is in production. Those come out of an architecture review or a threat model, which is cheaper than testing and is what testing should be aimed by.
The practical ordering: map the system, then test the parts of it that matter. Testing first produces findings nobody can place.
Why one phrase covers both, and the confusion that causes
The phrase arrived in AI from two directions at once. Security borrowed “red team” from adversary simulation, where it has meant objective-led attack for decades. Model developers borrowed the same phrase for structured probing of model behaviour, and regulators then used it in the broad sense.
Regulation (EU) 2024/1689 is a useful illustration of the broad usage. Its recitals state that providers of general-purpose AI models with systemic risk should conduct and document adversarial testing of models, also, as appropriate, through internal or independent external testing
, and that cybersecurity protection should consider accidental model leakage, unauthorised releases, circumvention of safety measures
and defence against unauthorised access or model theft
. Read that against the two disciplines above and it plainly reaches both: the first phrase sits closer to evaluation, the second squarely in security. The Regulation as published on EUR-Lex is the text to read, and whether any of it reaches a particular organization is a question for counsel.
The consequence in procurement is mundane and expensive. A buyer writes “red teaming” meaning penetration testing, receives an evaluation report, files it, and believes a question has been answered that was never asked.
Reading a vendor proposal
Three questions settle it, and none of them require you to understand the method.
- What is the target? If the answer names a model rather than an environment, a URL or a set of accounts, you are being sold evaluation.
- What counts as success? If success is a score, it is evaluation. If success is “we reached X”, it is security testing.
- What is in the report? Reproduction steps, affected component and a fix per finding is security testing. A table of rates is evaluation.
Usually the security one. A questionnaire line asking whether your AI has been red teamed is nearly always asking whether somebody competent attacked the deployed system and wrote down what they found, not whether the model scored well on a refusal benchmark. Most organizations need the security work and a much smaller amount of the evaluation work. Adversarial testing tells you what an attacker can reach through the system; evaluation tells you whether the model behaves acceptably on the inputs you expect, which matters most where the output reaches the public or drives a regulated decision.
How the work is bounded
The scope is agreed in writing before work starts, and the engagement is quoted in writing with it.
Testing and adversary simulation are carried out only with signed authorization, to a scope agreed in writing.
SecHB does not issue certifications, attestations or audit opinions: those come from accredited certification bodies, CPA firms and QSAs. The work here is what an organization does to be ready for them.
Nothing here is legal advice. Where a question turns on the law, the work is done alongside the client’s counsel, not instead of them.
Questions we are asked
Which does a customer questionnaire usually mean?
Usually the security one. A questionnaire line asking whether your AI has been red teamed is nearly always asking whether somebody competent attacked the deployed system and wrote down what they found, not whether the model scored well on a refusal benchmark.
Do we need both?
Most organizations need the security work and a much smaller amount of the evaluation work. Adversarial testing tells you what an attacker can reach through the system; evaluation tells you whether the model behaves acceptably on the inputs you expect, which matters most where the output reaches the public or drives a regulated decision.
Is a benchmark score evidence of security?
No. A benchmark score describes how a model responded to a fixed set of prompts under laboratory conditions; it says nothing about whether your retrieval layer leaks across tenants, whether a tool is wired in with more permission than it needs, or whether an uploaded document can redirect the assistant.
How do we tell which one a proposal is selling?
Three questions separate them in any proposal: what is the target, what counts as success, and what is the deliverable. Adversarial security testing targets a deployed system, succeeds by achieving an attacker objective, and delivers findings with reproduction steps; evaluation targets a model, succeeds by measuring behaviour against a rubric, and delivers scores.
Where to go from here
If what you need is the security side against a specific application, LLM application penetration testing is that engagement, and red teaming and adversary simulation is the broader objective-led form of it. If nobody has mapped the system yet, an AI security assessment comes first and makes the testing worth buying. The rest of the AI security practice sits behind those, and the other pieces are indexed under Writing.
Send the proposal and the questionnaire line that prompted it, and the reply will say which of the two you are being sold.