LLM red teaming: a step-by-step guide
How to run a structured red team against your LLM app — building attack cases, probing each vulnerability, scoring the results, and turning the whole exercise into a test suite you run forever.
TL;DR: Red teaming an LLM means deliberately trying to make it misbehave — jailbreaks, injections, leakage, harmful output — and recording what worked. Done as a one-off it's a report that goes stale in a week. Done right, it produces a scored adversarial dataset that you run on every model and prompt change and on live traffic, so a regression is caught instead of shipped.
Functional testing asks "does it work?" Red teaming asks "can I make it fail on purpose?" For LLM applications that second question is the one that matters for safety and security, because the failure modes are behavioral and cheap to trigger. This tutorial walks a red team from first attack to a permanent test suite.
Step 1: Define the threat model
Decide what "harm" means for your app before you attack it. A medical chatbot, a coding agent, and a customer-support bot have different worst cases. List the outcomes you must prevent — harmful content, data leakage, unauthorized tool actions, brand damage, hallucinated advice acted upon. Everything downstream targets these. (See the LLM security guide.)
Step 2: Build attack cases per vulnerability
For each risk, write attempts that probe it. Cover the known families:
- Jailbreaks — role-play, instruction override, obfuscation, fictional framing. (See how to jailbreak LLMs.)
- Prompt injection — payloads hidden in retrieved documents and tool output, not just typed by the user.
- Data extraction — attempts to pull PII, secrets, or another user's data.
- Harmful content — requests for disallowed output, direct and disguised.
- Excessive agency — inputs that try to push an agent into a tool action beyond its remit.
Draw from real production traffic where you can — the attacks users actually try beat the ones you imagine.
Step 3: Probe manually, then automate
Start by hand to find what works and understand why. Then automate the coverage: generate variations of each successful attack (rephrasings, encodings, longer multi-turn escalations) so you're testing a family, not a single string. Crescendo-style attacks only surface in multi-turn evaluation, so include conversations, not just single prompts.
Step 4: Score the results
An attempt isn't "pass/fail" by eye at scale — define a rubric. For each attack, the expected behavior is a refusal or a grounded, safe response, and the metric is guardrail-hold rate: the fraction of attempts the system correctly resisted. Open-ended judgments (was this response actually harmful?) are scored with LLM-as-a-judge against that rubric.
Step 5: Make it a permanent suite
This is the step that separates red teaming from a slide deck. Every attack you built and scored becomes a regression test:
- Run the adversarial suite on every model change — new versions regress on old attacks silently.
- Run it on every prompt change — reworded guardrails shift the balance.
- Run it against production traffic continuously, and fold new real-world attacks back into the suite. (See run LLM evals on production traces.)
Red-team checklist
| Phase | Output |
|---|---|
| Threat model | The harms specific to your app |
| Attack cases | Probes per vulnerability family |
| Automation | Variations + multi-turn escalations |
| Scoring | Guardrail-hold rate via rubric |
| Suite | Regression tests run on every change + production |
How Currai fits
Currai traces every attack attempt as the exact sequence that produced it — prompt, retrieved content, tool calls, response — so you can see how a successful attack got through, not just that it did. Score your adversarial dataset and live production traffic against a guardrail rubric, track hold rate across model and prompt changes, and promote any successful attack from a trace into a permanent test case. Red teaming stops being a one-off and becomes a continuous defense. See the LLM safety guide and secure enterprise deployment strategies, or start with Currai free.
Frequently asked questions
What is LLM red teaming?
Deliberately attacking your own LLM application — with jailbreaks, prompt injection, data-extraction attempts, and harmful-content requests — to find where it fails, so you can fix and continuously test those failure modes.
How is red teaming different from regular evaluation?
Regular evaluation measures quality on normal inputs; red teaming measures resistance to adversarial ones. Both should run continuously, but red teaming targets safety and security failures specifically.
Should red teaming be a one-time exercise?
No. A one-off is stale as soon as the model or prompt changes. Turn each attack into a scored regression test and run the suite on every change and against production traffic.
How do you measure red-teaming results?
With a rubric and a guardrail-hold rate — the fraction of attacks the system correctly resisted — using an LLM-as-a-judge to score open-ended cases like whether a response was actually harmful.