What Is LLM Red Teaming? A Practical Guide

Your LLM feature passes its demo, its unit tests, and the provider’s built-in safety filter. LLM red teaming is the discipline of finding where it breaks anyway — under adversarial input, against a real user trying to make it misbehave. It is a structured attack on the whole system you shipped, not a benchmark score on the model you called. If you own an AI feature that has to survive contact with real users or an enterprise security review, this is the surface you need to understand.

What is LLM red teaming?

LLM red teaming is adversarial testing of an LLM-backed system: the model plus everything you built around it — the system prompt, the retrieval pipeline (RAG), the tools and functions the model can call, the memory it keeps, and the downstream code that consumes its output. The goal is to find inputs and interaction sequences that make the system violate a trust boundary: leak data it should not, take an action it should not, or produce output that harms the business.

The key word is system. A raw model cannot spend money, email a customer, or read another tenant’s records — your integration can. The interesting failures live at the seams: where untrusted input meets a privileged tool, or where retrieved content is trusted as if it were an instruction.

Two properties separate real red teaming from a checklist:

• It is adversarial. The tester assumes an intelligent attacker with a goal, not a random fuzzer. Attacks are chained: one weak control alone is harmless; three of them composed become an incident.
• It is evidence-first. A finding is only real if it reproduces. Every finding should carry a request trace — the exact input, the model’s output, the tool call and its arguments — not a screenshot of a vibe.

How it differs from traditional penetration testing

Traditional application penetration testing and LLM red teaming overlap in mindset but target different layers. You need both; neither substitutes for the other.

The probabilistic row matters most: a payload that fails four times and lands on the fifth is still a live vulnerability — an attacker simply retries. Red teaming accounts for success rates, not binary pass/fail, and a useful report states how often an attack landed across attempts.

What gets tested

A thorough engagement works across four families of attack, then composes them into chains.

Prompt injection chains

Direct prompt injection is a user typing instructions that override your system prompt. The higher-impact variant is indirect injection: hostile instructions hidden inside content the model ingests — a support ticket, a fetched web page, a RAG document, an email it summarizes. The model cannot reliably tell your instructions from attacker instructions that arrived as data. This is LLM01 and the root of most serious findings.

Jailbreaks mapped to business risk

A jailbreak that makes the model say something rude is a demo. A jailbreak that matters is tied to a business consequence: bypassing a policy that gates a refund, extracting how to abuse your pricing, or producing content that creates compliance exposure. Red teaming ranks jailbreaks by the outcome they unlock, not by how surprising the output is.

Data exfiltration

Can a user pull out data the system was never meant to reveal? This spans the system prompt itself (LLM07 System Prompt Leakage), secrets in context, PII, and — worst case in a multi-tenant product — another customer’s data bleeding across the context boundary (LLM02 Sensitive Information Disclosure).

Tool and agent abuse

Once the model can call tools — issue credits, send messages, query databases, execute code, browse the web — the attack surface expands from words to actions. The tests probe excessive agency (LLM06): can crafted input drive the agent to call a tool beyond the user’s authority, with arguments the user should not control? For agentic systems this is where the OWASP Agentic Security Initiative and its Agentic Top 10 apply.

A worked attack chain (illustrative example)

The following is an illustrative example — a constructed scenario to show how individually minor weaknesses compose into a real incident. It is not a report of any real client system.

Consider an AI support agent for a SaaS product. It reads customer tickets, retrieves answers from a knowledge base, and can call two tools: read_customer_records and issue_account_credit.

1. Reconnaissance. The attacker opens a normal ticket and notices the agent fetches URLs mentioned in the ticket and references “checking your account” — so it ingests external content and holds account-level tools.
2. Weaponization. The attacker hosts a page whose visible text is a bland FAQ but which carries hidden text: an instruction telling any assistant that reads it to call issue_account_credit for 500 units and summarize the account’s recent tickets inline.
3. Delivery. The attacker submits a ticket linking that page. The agent fetches it into context as trusted content — indirect prompt injection (LLM01).
4. Exploitation. The injected instruction outranks the system prompt. The agent calls issue_account_credit (excessive agency, LLM06) and read_customer_records, returning ticket contents that belong to other customers (sensitive information disclosure, LLM02).
5. Impact. Unauthorized credits are financial loss; cross-customer data in the transcript is a privacy incident. Because each step looks like normal agent behavior, nothing obvious fires in monitoring.

No single control failed catastrophically. The system prompt was fine; the tools worked as designed. The failure is architectural: untrusted retrieved content was trusted as instruction, and a privileged tool sat one hop away with no authorization check between them. What makes the finding actionable is the request trace — the fetched page content in context, the exact tool call, and its arguments — which points an engineer straight at the missing boundary. That is the difference between a reproducible finding and a screenshot of a vibe.

What a useful red-team report contains

A report you can act on is structured around reproduction and remediation, not severity theatre. Every finding should carry:

• A stable identifier and a plain-language title — what breaks, in one line.
• The trust boundary crossed — data exfiltration, unauthorized action, policy bypass.
• A reproducible trace — the input, the output, the tool call and arguments, and the success rate across attempts.
• Framework mapping — OWASP LLM Top 10 ID plus the MITRE ATLAS technique, so it slots into your risk register.
• Business impact — the concrete consequence, in your terms, not a generic CVSS gesture.
• A remediation — the specific control that closes it, and how to test the fix holds.

Rendered as a structured object, a finding is easier to triage, dedupe, and re-verify than prose:

{
  "id": "PI-004",
  "title": "Indirect prompt injection via fetched ticket URL drives unauthorized tool call",
  "boundary": "unauthorized-action + data-exfiltration",
  "owasp_llm": "LLM01",
  "atlas": "AML.T0051 LLM Prompt Injection (indirect)",
  "success_rate": "6 of 10 attempts",
  "impact": "Agent issues account credit and returns other customers' ticket data",
  "remediation": "Isolate retrieved content from instruction context; gate issue_account_credit behind an out-of-band authorization check",
  "evidence": "trace-2026-...-full-request-response.json"
}

The evidence pointer is load-bearing. Findings without traces cannot be verified, cannot be regression-tested after the fix, and cannot survive an enterprise security reviewer asking “show me.” For a fuller worked example of report structure, see our sample assessment report at /resources/sample-ai-assessment-report.

Mapping attack classes to OWASP LLM Top 10 and MITRE ATLAS

This is the artifact to keep. Each row maps an attack class to its primary OWASP LLM Top 10 identifier and the business impact that makes it worth caring about. Use it to sanity-check that a proposed engagement covers your real exposure.

Layering MITRE ATLAS on top

The OWASP LLM Top 10 tells you what class of weakness you face. MITRE ATLAS tells you where in an attacker’s kill chain it sits, so you reason about detection and sequencing rather than isolated bugs. ATLAS organizes adversary behavior into tactics — reconnaissance, resource development, initial access, execution, persistence, exfiltration, impact — and names techniques under each. Prompt injection maps to the technique LLM Prompt Injection (AML.T0051), with direct and indirect variants; jailbreaks to LLM Jailbreak; data pulls to LLM Data Leakage; system-prompt extraction to Extract LLM System Prompt; corpus tampering to Poison Training Data; denial-of-wallet pressure to Denial of ML Service. Reporting against both frameworks lets a security team file the finding in a risk register they already trust and reason about it against NIST AI RMF’s govern, map, measure, and manage functions.

When to run an LLM red team

Run one when the cost of the system misbehaving becomes real:

• Before an enterprise security review. If a customer’s security team is about to assess your AI feature, find the LLM01 and LLM06 issues first, on your own terms, with fixes already in flight.
• Before granting the model tools or autonomy. The moment your model can spend, send, delete, or query, the impact of a successful injection jumps from embarrassing to material. Test before you ship the tools.
• After a material change. A new system prompt, RAG source, model version, or tool can silently regress a control that used to hold — a way red-team findings regress that traditional pentest findings usually do not.
• On a cadence in production. Continuous adversarial testing catches the drift a one-time engagement cannot.

Red teaming is one layer of production readiness, not the whole of it. It pairs with an evaluation suite that catches quality regressions and with the architectural controls — trust-boundary isolation, tool authorization, output handling — that actually close the findings. Our methodology at /methodology shows how adversarial testing, evals, and hardening fit into a single readiness picture, and the AI Product Readiness Assessment at /services/ai-product-readiness-assessment is the engagement that runs it end to end.

We reduce risk and prove exactly what we tested. We will not hand you a certificate that your AI is safe — no one honestly can. What a red team gives you is a shorter, evidenced list of the specific ways your system breaks and the specific controls that close each one.

To go deeper on the highest-frequency finding class, see prompt injection testing at /guides/prompt-injection-testing. For agent-specific attack surface — tool abuse, multi-step planning, and the Agentic Top 10 — see AI agent security at /guides/ai-agent-security.