Penetration test, vulnerability scan, automated pentest: which does your contract mean?
A contract says you must have an annual penetration test. A quote arrives that costs a tenth of another quote. Both suppliers use the phrase. One is selling a scheduled scan, one is selling a platform that replays known attack patterns, and one is selling several weeks of a person trying to break your application. All three are legitimate products. Only one of them is what the clause means.
The short version
- Vulnerability scanning compares what you are running against a database of known issues. It is fast, cheap, repeatable and should be running all the time.
- Automated penetration testing platforms go further: they attempt exploitation, chain a limited set of known techniques, and confirm that a finding is real rather than theoretical.
- Human-led penetration testing is a person reasoning about your specific system — its roles, its business rules, its assumptions — and trying to make it do something it was built not to do.
The gap between them is not effort. It is the class of problem each one can see at all.
Vulnerability scanning: what it does and cannot do
A scanner fingerprints what is there and matches it against published weaknesses: an unpatched service, a library with a known issue, a certificate about to expire, a default configuration left in place, a port that should not be open. It produces a list, usually a long one, with severities drawn from a published scoring system rather than from your environment.
What it cannot do is understand what your application is for. It does not know that a user in one organization should never see a record belonging to another, because that rule exists only in your code and your intentions. Under the OWASP Top 10:2025 that puts the first category — broken access control — largely out of a scanner’s reach, along with insecure design and most of the new category covering the mishandling of exceptional conditions. It is strong on security misconfiguration and on the dependency half of software supply chain failures.
A scanner is not a weak penetration test. It is a different instrument, and an organization without continuous scanning is failing at something a human tester should never have been spending a week on.
Automated testing platforms: where they help and where they stop
These sit between the two. They attempt exploitation rather than inferring it from a version number, which removes a large share of the false positives that make scan output tiring to read. Against internal networks in particular they are genuinely strong: credential reuse, weak service configurations, privilege escalation paths and lateral movement through Windows environments are well-trodden ground and can be automated convincingly.
They stop where the technique library stops. An automated platform reaches the attacks somebody has already encoded. It does not read your pricing logic and notice that a discount code validated on the client is trusted on the server. It does not observe that two endpoints check the same permission in two different ways and that one of them was written before your tenancy model changed. Those findings come from somebody forming a hypothesis about your system, which is still a human activity.
Used well, a platform runs between human engagements and keeps the floor from sinking while nobody is looking.
Human-led testing: chains, logic and the attack path
The distinguishing output of a human-led test is not a longer list. It is usually a shorter one, with a narrative attached: this low-severity information disclosure gave us a valid internal identifier, which this endpoint accepted without checking ownership, which returned a token with a scope nobody intended, which let us do the thing on the last page of the report.
No individual step in that chain would rate as critical on its own, and no scanner would have joined them. Business-logic findings behave the same way: they are not defects in a library, they are correct implementations of a rule that turns out to be wrong. Finding them requires understanding what the application is supposed to prevent, which requires being told.
This is also the kind of testing that carries the most obligations around it: a scope agreed in writing, an authorization signed by someone who can give it, and rules of engagement covering what happens if something breaks. Our own testing work is described under penetration testing.
What each one reaches
| Class of problem | Scan | Automated platform | Human-led test |
|---|---|---|---|
| Known unpatched components | Strong | Strong | Confirms, rarely the focus |
| Security misconfiguration | Strong | Strong | Strong, in context |
| Credential and privilege paths | Partial | Strong | Strong |
| Access control between tenants or users | Rarely | Partial | Strong |
| Business logic and pricing rules | No | No | Strong |
| Chained multi-step attack paths | No | Limited | Strong |
| Design flaws and failure-open behaviour | No | Limited | Strong |
| Runs continuously | Yes | Yes | No |
Read the last row with the others. The activity that finds the most is the one that runs least often, which is the argument for having all three rather than choosing between them.
What does a compliance clause usually mean?
Read the clause for three words: who performs the work, what method is named, and what artifact has to be delivered. Those three decide which of the three activities satisfies it, and a clause that names none of them is worth clarifying in writing before you buy anything.
Payment card work is the clearest example of a regime that answers the question for you, because the programme names who may perform the assessment: the PCI Security Standards Council qualifies and lists Qualified Security Assessors, and a validation performed by anyone else is not that validation. Most other clauses are far vaguer, which is where the confusion lives. A customer contract that says “annual penetration testing by a qualified third party” is asking for the human-led activity and an artifact they can read; a clause that says “regular vulnerability scanning” is asking for the scheduled one.
When the clause genuinely is ambiguous, ask the counterparty which artifact their reviewer expects to receive. It is a short email and it has settled more of these than any amount of interpretation.
What an insurer usually means
Cyber insurance applications tend to ask two separate questions — whether you scan, and whether you test — precisely because the underwriter knows they are different. The answers are read together with the controls questions around them, and an answer that overstates what you have is the answer that causes trouble at claim time rather than at renewal.
If you scan continuously and test annually, say exactly that. If you run an automated platform and have never had a human-led engagement, say that too; it is a defensible position for many organizations and a far better answer than one that cannot be evidenced.
Reading a quote: four tells
- Named methodology. A human-led engagement names one — the OWASP Web Security Testing Guide for web applications, or an equivalent for the platform in question. A scan quote names a product instead.
- Effort expressed in time. Human testing is quoted in days of work against a defined scope. If nothing in the quote is measured in days, nobody is planning to spend any.
- Credentials and roles. A quote that does not ask which user roles exist is not planning to test between them.
- The deliverable. Ask for a redacted sample. A report with reproduction steps and an attack-path narrative is a different product from an exported findings list, and what a single finding looks like written up properly is the fastest way to see the difference.
A scanner export submitted as a penetration test report is the single most common thing a security reviewer rejects, because it has no scope statement, no reproduction steps and no evidence that a human tried to chain anything. The sections a real report carries are set out in what a penetration test report should contain.
Getting value from all three
Scanning belongs on a schedule and in the pipeline — continuously for infrastructure, on every build for dependencies. Human-led testing belongs at the points where the system changes shape: a new product, a new trust boundary, a new tenant model, or a customer requirement. Scanning holds the floor. A platform keeps the internal network honest between engagements. A person is pointed at the part of the system where being wrong would cost the most.
The sequence matters as much as the mix. Running a human-led test over an environment that has never been scanned wastes the expensive activity on findings the cheap one would have caught, which is a common and entirely avoidable way to spend a testing budget. Clear the known issues first, then aim the people at the unknown ones — and feed what they find back into how the software gets built so the same class does not return next year. Scoping the engagement itself is covered in how to scope a penetration test.
How the work is bounded
The scope is agreed in writing before work starts, and the engagement is quoted in writing with it.
Testing and adversary simulation are carried out only with signed authorization, to a scope agreed in writing.
SecHB does not issue certifications, attestations or audit opinions: those come from accredited certification bodies, CPA firms and QSAs. The work here is what an organization does to be ready for them.
Nothing here is legal advice. Where a question turns on the law, the work is done alongside the client’s counsel, not instead of them.
Questions we are asked
Is a vulnerability scan ever enough?
A scan is enough when the question is whether you are exposed to a known published weakness, and it is never enough when the question is whether your own logic can be abused by someone holding a valid account.
How often should each one happen?
Scanning belongs on a schedule and in the pipeline — continuously for infrastructure, on every build for dependencies. Human-led testing belongs at the points where the system changes shape: a new product, a new trust boundary, a new tenant model, or a customer requirement.
Can a scan report be submitted as a penetration test report?
A scanner export submitted as a penetration test report is the single most common thing a security reviewer rejects, because it has no scope statement, no reproduction steps and no evidence that a human tried to chain anything.
How do we tell what our contract clause means?
Read the clause for three words: who performs the work, what method is named, and what artifact has to be delivered. Those three decide which of the three activities satisfies it, and a clause that names none of them is worth clarifying in writing before you buy anything.
Working out which one you need
Send the clause and the quote. The reply says which of the three you are being sold and whether it satisfies the obligation. Related pieces are indexed at writing.