Cybersecurity assessment vs penetration test: what buyers mean

The same-sounding request tends to arrive from more than one direction at once. Security questionnaires ask whether a company performs annual penetration testing. Insurance applications ask for the results of a recent security assessment. Boards ask for the company to be assessed. The words overlap enough that it is tempting to buy a single engagement and assume it answers all of them.

It usually does not. These are two different exercises with different outputs and different readers, and buying one when the question in front of you needed the other is the most reliable way to spend a security budget without closing anything.

The two words answer different questions

A penetration test answers: can someone get in, and how far do they get?

A cybersecurity assessment answers: do we have the right controls, are they designed properly, and are they operating?

That is the whole distinction, and almost every downstream confusion follows from ignoring it. A clean penetration test report tells you nothing about whether your offboarding process works. A clean assessment tells you nothing about whether your login page can be bypassed.

What the standard-setters actually call them

The vocabulary is more settled than the market makes it sound.

NIST's glossary gives penetration testing as "a test methodology in which assessors, typically working under specific constraints, attempt to circumvent or defeat the security features of a system," and records the NIST SP 800-115 formulation as "security testing in which evaluators mimic real-world attacks in an attempt to identify ways to circumvent the security features of an application, system, or network." Both definitions are about attempting, circumventing and mimicking. It is an adversarial exercise.

"Cybersecurity assessment" is the looser term, and the closest formal cousin is security control assessment, which NIST's glossary defines as "the testing and/or evaluation of the management, operational, and technical security controls in an information system to determine the extent to which the controls are implemented correctly, operating as intended, and producing the desired outcome with respect to meeting the security requirements for the system."

The useful part is the next definition along. An assessment method, in NIST's glossary, is "one of three types of actions (i.e., examine, interview, test) taken by assessors in obtaining evidence during an assessment."

Read that carefully: test is one of three methods inside an assessment. A penetration test is not the opposite of an assessment. It is one technique an assessment can use, and the narrowest of the three.

The third term people mix in is vulnerability assessment, which NIST's glossary gives as a "systematic examination of an information system or product to determine the adequacy of security measures, identify security deficiencies, provide data from which to predict the effectiveness of proposed security measures, and confirm the adequacy of such measures after implementation." In practice that usually means scanning. A scan produces a list of things that look wrong. A penetration test tries to prove which of them actually are.

Where a penetration test is genuinely required

In our experience, most companies asking this question are not subject to a rule that names a penetration test at all. The demand is coming from a customer contract, an insurance application or a questionnaire — which is a commercial requirement, and negotiable in a way a rule is not.

Where a rule does name it, the FTC's Safeguards Rule is the clearest example to point at. The rule defines penetration testing as "a test methodology in which assessors attempt to circumvent or defeat the security features of an information system by attempting penetration of databases or controls from outside or inside your information systems."

The FTC's own guidance on the rule sets out the testing obligation: covered financial institutions must conduct annual penetration testing, as well as vulnerability assessments — described as "system-wide scans every six months designed to test for publicly-known security vulnerabilities" — unless they take the alternative route, because "for information systems, testing can be accomplished through continuous monitoring of your system." The guidance also frames the whole thing as ongoing rather than one-off, requiring reassessment "in light of changes to your operations or the emergence of new threats."

Two things are worth taking from that even if the Safeguards Rule does not apply to you. First, the regulator treats the penetration test and the vulnerability assessment as different obligations at different intervals, not as synonyms. Second, continuous monitoring is treated as capable of doing the same job — which tells you the point was never the report.

Whether you are a covered institution, and which paragraphs apply to you, is a question for counsel rather than for a vendor's proposal.

What each one produces

	Penetration test	Cybersecurity assessment
Core question	Can we be broken into, and how	Are the controls right and working
Method	Attempted exploitation	Examine, interview and test
Output	Attack paths, proof of exploitation, severity ratings	Control-by-control findings and a gap list
Time reference	The days it was performed	Design now, and often operation over a period
Best reader	Engineering	The board, the auditor, the buyer
Fails when	Scope was too narrow to reach anything interesting	It never touches evidence, only policy documents

The government's own framing is a useful sanity check on what a test is for. The federal Penetration Testing shared service listed by CISA is described as helping agencies "use a variety of tactics, techniques, and procedures to identify exploitable vulnerabilities in networks and systems," and as measuring "compliance with organizational security policies by detecting whether staff are aware of security issues and, ultimately, determining the organization's risk of cybersecurity threats." Exploitable is the operative word. Not present, not misconfigured — exploitable.

The failure mode in each direction

Buying a test to answer a control question. Where the question is how privileged access is granted, reviewed and removed, a penetration test report cannot close it, however good the test was. That is a control question, and it is answered with a population, a reviewer and a record — the ground covered in how to run an access review an auditor will accept. Sending the wrong artefact leaves the review open and the budget spent.

Buying an assessment to answer a test question. Where a contract names a penetration test by an independent party, a readiness assessment does not satisfy the clause, however thorough it is. Read the clause before scoping anything.

Buying either one without a scope in writing. A test scoped to a marketing site and a staging environment will come back clean, and it will be worthless. An assessment scoped to policy documents will come back tidy, and it will be worse than worthless, because it creates a record saying you looked.

Questions that decide whether the money is well spent

1. Who is going to read the result, and what decision do they make with it? If nobody can name the reader, do not buy yet.
2. What exactly is in scope, written down before work starts? Systems, environments, accounts, and whether social engineering is included.
3. Is there a retest after remediation? A report with no retest describes a company that no longer exists.
4. Is the assessment going to look at evidence, or only at documents? The difference between a useful gap list and an expensive summary of your own policies.
5. What is the report allowed to be shared as? Some customers want the full report; many testing firms will only permit a summary letter. Settle it before signing.

Where to start

If you are being asked for both and cannot do both this quarter, our view is that the assessment comes first in most cases. It tells you where the weaknesses are before you pay someone to demonstrate them, and it produces the artefact that more reviewers can actually read. The exception is a contractual deadline that names a penetration test; then the deadline decides.

If a questionnaire is what triggered this, the security questionnaire panic checker will sort the questions that need a document from the ones that need a control to exist. If you are trying to work out which gaps are worth closing first, the readiness scorecard will give you an ordered list. And if the driver is a buyer rather than a regulator, SOC 2 Type 1 vs Type 2 is the more useful next read, because the report they want is probably not a penetration test at all.

This is general information, not audit, security or legal advice. Confirm what applies to you with your auditor or counsel.

Common questions

Is a penetration test the same thing as a vulnerability scan?

No. NIST's glossary describes a vulnerability assessment as a systematic examination to determine the adequacy of security measures and identify security deficiencies, while penetration testing is a test methodology in which assessors attempt to circumvent or defeat a system's security features. A scan lists what looks wrong; a penetration test tries to prove it.

Does any regulation actually require an annual penetration test?

Some do, for the entities they cover. The FTC's guidance on its Safeguards Rule says covered financial institutions that do not implement continuous monitoring must conduct annual penetration testing as well as vulnerability assessments. Whether you are a covered financial institution, and which paragraphs apply to you, is a question for counsel.

Will a penetration test satisfy an auditor or a customer's security review?

Sometimes it is one input among several, and it is rarely the whole answer. A test report shows what an attacker could reach on given dates; a reviewer usually also wants evidence that access reviews, change management and monitoring were operating across a period. Ask the requester which question they are trying to close.

Which should a company buy first?

In our view, an assessment first in most cases, because it tells you where your controls are weak before you pay someone to prove it. The exception is a customer or insurer with a contractual deadline that names a penetration test, in which case the deadline decides the order.

What makes a penetration test report useful rather than decorative?

A defined scope written down before the work starts, an agreed rules-of-engagement document, findings that name the path taken rather than only the tool output, and a retest after remediation. Without the retest, the report only ever describes the state you were in before you fixed anything.

How often should either one be repeated?

The interval should follow your rate of change, not the calendar alone. Annual is the cadence most customer contracts and questionnaires we see name, which is a commercial convention rather than a universal rule, and a material change to your architecture, your hosting or your customer data flows is a better trigger than a date.

Sources

1. FTC Safeguards Rule: What Your Business Needs to Know · Federal Trade Commission
2. Standards for Safeguarding Customer Information; Final Rule, 86 FR 70272 (9 December 2021) — see the definition of penetration testing at 16 CFR 314.2 · Federal Trade Commission, via the U.S. Government Publishing Office
3. Penetration Testing — federal shared service description · Cybersecurity and Infrastructure Security Agency
4. Computer Security Resource Center Glossary — 'assessment method' (definitions drawn from NIST SP 800-53A Rev. 5 and NIST SP 800-137A) · National Institute of Standards and Technology
5. Computer Security Resource Center Glossary — 'security control assessment' (definition drawn from NIST SP 800-137) · National Institute of Standards and Technology
6. Computer Security Resource Center Glossary — 'penetration testing' (definitions drawn from NIST SP 800-53 Rev. 5, NIST SP 800-115 and others) · National Institute of Standards and Technology
7. Computer Security Resource Center Glossary — 'vulnerability assessment' (definitions drawn from CNSSI 4009-2022 and NIST SP 800-137) · National Institute of Standards and Technology