AI Red Teaming: How Security Teams Test Whether an AI System Can Be Exploited

Traditional security testing was built for software that behaves the same way every time. The same input, the same output, every time — which means a vulnerability, once confirmed, can be reliably reproduced, documented, and fixed. AI systems don't work that way. They generate probabilistic responses, process instructions and data through the same channel, and can be manipulated in ways no conventional penetration testing methodology was designed to surface.

 

AI red teaming exists to fill that gap. It's systematic adversarial testing — deliberately attempting to exploit an AI system's behavior, its integrations, its surrounding architecture, and the trust assumptions baked into how it was deployed. Done well, it answers the question most organizations haven't formally asked yet: not whether their AI system behaves correctly under normal conditions, but whether it can be made to behave incorrectly under adversarial ones.

 

This piece covers how that actually works in practice, using the frameworks and tooling security teams are using in 2026.

 

 

The Three Frameworks That Structure the Work

AI red teaming isn't improvised. It's built on a small number of frameworks that give engagements structure, comparability, and — critically — a shared vocabulary that makes findings actionable rather than anecdotal.

 

1. MITRE ATLAS

MITRE ATLAS (Adversarial Threat Landscape for AI Systems) is the discipline's equivalent of the ATT&CK framework used across conventional cybersecurity. As of its v5.1.0 release in November 2025, ATLAS covers 16 tactics, 84 techniques, 32 mitigations, and 42 documented case studies drawn from real-world AI attacks — not theoretical scenarios. A February 2026 update added further techniques including "Publish Poisoned AI Agent Tool" and "Escape to Host," reflecting how quickly the threat landscape is evolving. Critically, approximately 70% of ATLAS mitigations map to existing security controls, which means organizations with mature traditional security programs have more applicable coverage than they typically realize.

 

2. OWASP Top 10 for LLM Applications

OWASP Top 10 for LLM Applications provides the vulnerability classification scheme most widely used in LLM-specific red teaming. In its 2025 edition, Prompt Injection holds the LLM01 position — the top risk to LLM implementations — a position it has held since the list's inception, because the structural problem it describes (LLMs processing instructions and data through the same channel) remains fundamentally unresolved. The 2026 OWASP Agentic Top 10 extends the taxonomy specifically to autonomous AI agents.

 

3. NIST AI 100-2 E2025

NIST AI 100-2 E2025 (the adversarial machine learning taxonomy, updated March 2025) and its companion NIST AI 600-1 (the GenAI Profile of the AI Risk Management Framework) provide the compliance-anchoring layer. NIST AI 600-1 specifically recommends red teaming before and after deployment across 12 GenAI risk categories. These frameworks don't tell you how to run a test — as one practitioner guide puts it plainly: "governance documents tell you that testing should happen and roughly what to worry about, they do not tell you how to run a test." What they provide is the structure for proving that a program exists and for mapping findings in a way that satisfies auditors and boards.

 

Using all three in combination is the current industry standard. ATLAS for naming what was tested. OWASP for the vulnerability taxonomy. NIST for the governance and compliance layer.

 

 

The Tooling: Garak and PyRIT Do Different Jobs

The two tools most widely used in AI red teaming today solve different problems, and understanding the difference is important for scoping what an engagement actually involves.

 

1. Garak (NVIDIA's open-source LLM vulnerability scanner) is the right first pass. You point it at a model endpoint, run its probe families — covering injection, jailbreak, data leakage, and other vulnerability classes — and get a report. It's designed to run at scale, it can be integrated into CI/CD pipelines for continuous coverage, and it handles the breadth of known vulnerability classes efficiently. For organizations that want ongoing automated coverage of a production model, Garak is the baseline.

 

2. PyRIT (Microsoft's Python Risk Identification Toolkit) handles a different category of attack: the ones that require state. Multi-turn conversations, an attacker model driving probes dynamically, scoring logic customized to the specific system being tested. Named strategies include Crescendo (gradually escalating multi-turn manipulation), TAP (Tree of Attacks with Pruning, which iteratively refines attack prompts), and Skeleton Key (an approach that attempts to remove a model's safety behaviors entirely through multi-step dialogue). PyRIT supports text, audio, image, video, and file modalities — and works against OpenAI, Azure, Anthropic, Google, and custom endpoints.

 

The practical takeaway: Garak for breadth and automation; PyRIT for depth and multi-turn, stateful attacks. Most mature red team programs use both, with Garak in the pipeline and PyRIT for the manual, expert-driven investigation that follows. Neither replaces human judgment about which attack vectors matter most for a specific system and deployment context.

 

Additional tools in active use: Promptfoo (133 built-in plugins with OWASP and MITRE mapping), AgentDojo (629 agent hijacking test cases from ETH Zurich), and Microsoft's Counterfit and IBM's Adversarial Robustness Toolbox for traditional ML models.

 

 

What the Five-Phase Assessment Actually Looks Like

A structured AI red team engagement follows a sequence that mirrors traditional penetration testing in its logic, even where the specific techniques are entirely different.

 

Phase 1: Scoping and threat modeling. 

Before a single test is run, the engagement needs a clear definition of what's being tested, under what access conditions, and against what threat model. Who would realistically attack this system? What are they after — data extraction, behavior manipulation, system compromise, reputational damage? What access do they realistically have — public API, authenticated user access, insider access, or black-box access only? The answers shape everything that follows. A black-box engagement against a public-facing chatbot requires different techniques than a white-box assessment of an internally deployed AI agent with access to enterprise systems.

 

Phase 2: Reconnaissance. 

Understanding the system's behavior before attempting to break it. What model is being used, or can it be inferred? What does the system prompt appear to contain? What tools and integrations does the system have access to? What data can it retrieve? What does the output format reveal about the underlying architecture? Many vulnerabilities become visible only once the system's normal behavior is mapped well enough to recognize deviations from it.

 

Phase 3: Automated vulnerability scanning. 

Running Garak probe families and Promptfoo plugins against the target to establish broad coverage across known vulnerability classes — prompt injection variants, jailbreak resistance, data leakage, denial of wallet conditions, and model behavior inconsistencies. This phase surfaces the low-hanging fruit and establishes a baseline that the manual phase can build on.

 

Phase 4: Manual adversarial testing. 

This is where the engagement shifts from scanning to genuine red teaming. Human testers use PyRIT's orchestration capabilities and direct interaction to test attack vectors that automated tools weren't built to find: multi-turn manipulation sequences, indirect prompt injection via documents or retrieved content, agentic tool misuse, goal hijacking, cross-session persistence, and attack chains that combine multiple individually minor weaknesses into a meaningful exploit path. As one practitioner noted after executing these attacks in a lab environment: "the most dangerous attack vectors are the simplest ones, because the LLM handles the exploitation complexity for the attacker. Every individual tool call in these attacks was legitimate. The malicious intent existed only in the sequencing."

 

Phase 5: Findings documentation and ATLAS mapping. 

Every finding is documented with the exact input used, the model's response, the rate of reproducibility across multiple attempts, and the ATLAS technique ID and OWASP category that describes it. Reproducibility documentation matters specifically because AI outputs are probabilistic — a finding that occurred once in twenty attempts carries a different risk profile from one that occurs in eighteen of twenty, and a report that doesn't quantify this is producing incomplete risk information.

 

 

The Category Most Engagements Underweight: Agentic Systems

The fastest-evolving area in AI red teaming in 2026 is autonomous agents — AI systems that take actions rather than just generate text. MITRE ATLAS's October 2025 collaboration with Zenity Labs added 14 new attack techniques specifically for AI agents, and the January 2026 v5.3.0 update added three new case studies covering MCP server compromises, indirect prompt injection via MCP channels, and malicious AI agent deployment.

 

Agentic systems change the red teaming picture significantly. A chatbot that generates bad text is a bounded harm. An agent with access to email, calendar, file systems, and API integrations that can be manipulated into taking unauthorized actions is a fundamentally different risk profile. Testing agentic systems requires specifically evaluating whether an agent can be induced to take actions outside its intended scope through crafted inputs, poisoned tool outputs, or malicious content embedded in retrieved documents — none of which conventional red teaming was designed to test for.

 

 

What a Regulatory Environment Demanding This Looks Like

AI red teaming is no longer purely voluntary in markets that matter. The EU AI Act requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing — obligations that have been in force since August 2, 2025, with full compliance for high-risk AI systems required by August 2, 2026. Penalties for non-compliance reach up to €35 million or 7% of global annual turnover.

 

More broadly, one in four malicious breaches is now AI-enabled — up 56% year-over-year — and breaches involving AI cost an average of $6 million against a $4.99 million global average. More than 20% of organizations have reported breaches targeting their AI models or applications specifically.

 

The combination of regulatory pressure and demonstrated financial exposure is what has moved AI red teaming from a practice that forward-thinking organizations do voluntarily to one that any organization deploying AI at meaningful scale needs to have a documented answer for.

 

 

What This Means for Your Security Program

The practical starting point for any organization that hasn't yet run a formal AI red team engagement is the same regardless of what AI systems are deployed: understand the threat model first. Who would attack this system, with what access, and what would they be trying to achieve? That question shapes whether a Garak scan is sufficient coverage, whether PyRIT-based multi-turn testing is warranted, whether the agentic-specific techniques need to be in scope, and what the findings need to demonstrate for the compliance and governance requirement that increasingly exists in the background.

---

Comments

---

No Comments Found.