How CISOs Should Scope Their First AI Red Team Engagement

The riskiest sentence in an AI red team brief is "just test our chatbot."

 

It sounds like a scope. It is not one. Does "the chatbot" include the documents it reads, the systems it can call, the model provider behind it, or the staff member who can change its instructions? Each answer changes what gets tested, how long it takes, and whether the report is any use to you afterwards. Good AI red team scoping is mostly the work of replacing that sentence with something specific, and it is far cheaper to do before the engagement than during it.

 

 

Start from what would hurt

Resist the urge to begin with attack techniques.

Microsoft's AI Red Team, says it has red-teamed more than 100 generative AI products, and has described a shift from model-only testing toward evaluating full systems, and one lesson it highlights is to start from downstream impact, not attack mechanics.

 

In practice, write three to five sentences beginning "This must never happen." For example:

The assistant must never show one customer another customer's data.

It must never take a financial action without human approval.

It must never reveal internal instructions or credentials.

 

These sentences become the spine of the engagement, and the final report can be read against them.

 

 

Pick one system, and pick it on purpose

For a first engagement, choose a single system. The usual candidates are the one customers see, the one that handles personal data, or the one that can take actions on someone's behalf. If a system does all three, that settles it.

 

Bundling five assistants into one brief to "get it all done" is tempting. Depth beats breadth the first time, because you also learn how an engagement works, what the report looks like, and how your teams handle findings.

 

 

Draw the whole system, not just the model

OWASP's GenAI Red Teaming Guide describes four areas: model evaluation, implementation testing, infrastructure assessment, and runtime behaviour analysis. A CSO Online summary adds that OWASP suggests scoping around lifecycle phases, risk scope (model, infrastructure, ecosystem) and risk source. That is a lot of framework for a first-time buyer, so here is the plain version. Ask your team to sketch, on one page:

 

* the model, and who hosts it

* the instructions and rules it is given

* every source of content it reads: documents, emails, web pages, databases

* every tool or API it can call, and the permissions each one carries

* where data goes, including to outside providers

* what is logged, and who looks at it

 

Anything on that page can be in scope. Anything you leave off will not be tested, so leave things off deliberately.

 

 

Decide who the attacker is

"A hacker" is not a threat model. Pick two or three from this list, because each needs different tests:

 

* an anonymous visitor typing into the public interface

* a logged-in customer trying to reach someone else's data

* an employee or contractor with legitimate access

* someone who can plant content the AI will later read, such as a support ticket, a shared file or an email

 

The last one deserves a place in most scopes, because it covers attacks where the AI is steered by content, not by the person talking to it. If your system has agents, the Cloud Security Alliance's Agentic AI Red Teaming Guide lists 12 high-risk threat categories, including goal manipulation, authorization and control hijacking, and bypassing human oversight. It is a useful checklist for what an agent engagement should cover.

 

 

Set the safety rails

This is the part first-time buyers skip, and it is where engagements go wrong. Settle these before day one:

 

* Where testing happens. Staging is safer. Production is more realistic. If production, agree which actions are forbidden.

* What data is used. Test accounts and synthetic records, wherever possible.

* What cannot be touched. Payment execution, outbound messages to real customers, and third-party systems you do not own.

* Your model provider's terms. Check what the provider allows in testing, and tell them if required.

* A stop button. One named person at your end who can pause the test, reachable during testing hours.

* A contact path for serious findings. Anything critical should not wait for the final report.

 

 

Humans and tools: ask what the mix is

Microsoft's team reports that automation helps with generating prompts, orchestrating attacks and scoring responses, but that red teaming cannot be fully automated, since people are needed for subject expertise, cultural context and judgement. Take that as a prompt to ask any provider: what share of this engagement is automated sweep, and what share is a person thinking about your specific system? There is no single right answer, but there is a wrong one, which is not knowing.

 

 

Insist on retest and a usable report

Put these in the scope, in writing:

* findings with steps someone on your team can repeat

* a severity rating tied to your "must never happen" sentences

* clear fix guidance, owned by named roles on your side

* a retest after fixes, included in the price

 

Microsoft describes break-fix cycles, meaning repeated rounds of testing, measurement and mitigation, as part of how it keeps AI systems safe. A first engagement that ends without a retest leaves you with a list and no proof anything changed.

 

 

A one-page scoping brief

If your team needs a starting point, use this:

1. System:              [name, owner, business purpose]
2. Must never happen:   [3–5 statements]
3. In scope:            [model, prompts, data sources, tools, integrations]
4. Out of scope:        [list, with reasons]
5. Attackers:           [2–3 personas]
6. Environment:         [staging / production, accounts, data]
7. Safety rails:        [forbidden actions, stop contact, escalation path]
8. Approach:            [automated vs manual, as stated by the provider]
9. Deliverables:        [report, evidence, severity scheme, retest]
10. Decision owner:     [one named person who accepts the risk]

 

That last line matters. Findings without an owner who decides what to fix tend to sit in a document.

 

 

Where ILLUME Intelligence can help

ILLUME Intelligence can help you draft that scope before you commit to anything, then run the engagement through our AI Red Teaming service, with our AI Penetration Testing (AI VAPT) and LLM Security Testing services covering the application layers around the model. Our vCISO service can take ownership of what happens after the report lands. If you are not sure what to put on the page, bring the system diagram and we will start there.

 

 

The Nutshell 

Choose one system.

Say what must never happen.

Draw the whole thing, name the attacker, set the safety rails, and agree on a retest.

If you can fill in that brief, you are ready to buy.

---

Comments

---

No Comments Found.