Most prompt injection testing fails for the same reason: someone points a scanner at an endpoint, gets a report full of green, and files it. The scanner was probably fine. What was missing was a definition of what counts as a failure in this application, a corpus that reflects this threat model, and a metric that survives being run twice.
Prompt injection testing is not a tool choice. It is six decisions, and the tool is the fifth-most important of them. What follows is a method you can run against a chatbot, a RAG pipeline or an agent, structured so the output is a tracked number rather than a pass/fail screenshot. It pairs with the framework-level comparison in Garak vs. PyRIT vs. promptmap — that piece tells you which harness to run, this one tells you what to do around it. For a condensed field version of the steps below, the prompt injection testing checklist on aisec.blog is a ready-made list to adapt to your own surfaces.
Step 1: Write the failure assertion before you write the test
“Is my app vulnerable to prompt injection?” has no testable answer. “Can retrieved content cause the agent to call send_email with a recipient the user did not name?” does.
Write one assertion per consequence you actually care about. Good assertions name a channel, an actor and an observable effect:
- Injected content in a retrieved document causes the system prompt to appear in the response.
- Injected content causes the model to emit a URL containing conversation text.
- Injected content causes any write-scoped tool to fire without a user confirmation step.
- Injected content causes the assistant to return a fixed attacker-chosen string instead of an answer.
Anything you cannot phrase this way, you cannot score. Anything you can, you can regression-test forever.
Step 2: Enumerate the injection surfaces
The assertion list depends on which channels exist in your architecture. Direct injection surfaces are the user turn and any user-supplied file. Indirect surfaces are everything the pipeline pulls in on its own: retrieved chunks, fetched web pages, email bodies, tool responses, repository files, image and document metadata. The channel inventory and what each one enables is covered in indirect prompt injection: how these attacks work.
If you would rather derive the list mechanically than from memory, the injection threat modeler takes your application’s building blocks and returns the attack classes those blocks make reachable, along with the trust boundary each one crosses. That output is a serviceable first test plan — one assertion per reachable class.
Step 3: Choose a corpus, and include one you did not write
Test corpora come from three places, and a serious plan uses all three.
Published benchmarks give you comparability and stop you from only testing attacks you already thought of. BIPIA covers indirect injection against models consuming external content. InjecAgent covers tool-integrated agents with 1,054 test cases across 17 user tools and 62 attacker tools. AgentDojo provides a dynamic environment where attacks and defenses are both evaluated against live task suites — 97 realistic tasks across email, e-banking and travel booking, and 629 security test cases.
Harness-generated probes come from your scanner’s own probe library — garak ships dozens of vulnerability-class probes with matching detectors, and PyRIT generates and mutates attack prompts through configurable orchestrators.
Application-specific payloads are the ones only you can write, because they name your tools, your document formats and your system prompt’s actual phrasing. This is usually where the findings are.
Two corpus properties matter more than size. Include encoded and obfuscated variants — a filter that catches ignore previous instructions in plain ASCII may pass the same string in Unicode Tag characters, and a corpus without them overstates your coverage. And include an adaptive round: take the payloads that failed, rephrase them against whatever blocked them, and re-run. Non-adaptive testing measures your filter’s memory, not your application’s resilience.
Step 4: Pick the harness to match the layer under test
| Layer under test | What you are asking | Fit |
|---|---|---|
| Base model, no application context | Does this model exhibit injection-vulnerable behavior at all? | garak — broad probe library, built-in detectors, good as a model-selection gate |
| Deployed application, system prompt and all | Does this configuration resist attacks against its rules? | promptmap for system-prompt rule testing; garak against the app endpoint |
| Custom multi-turn or multi-modal campaigns | Can an orchestrated, mutating attack chain get through? | PyRIT — orchestrators, converters, scorers, built for campaign design |
| Agent with tools and state | Can injected content cause a harmful tool call? | AgentDojo or InjecAgent — the only layer that scores utility and security together |
The common mistake is running a model-level scanner and reporting the result as an application finding. A clean garak run against gpt-4o says something about the model. It says nothing about your retrieval layer, your tool scopes, or the fact that your prompt assembler concatenates untrusted chunks with no separator. Match the harness to the layer or the report is measuring the wrong system.
Step 5: Define the detector, because this is the hard part
Firing payloads is easy. Deciding whether the response constitutes a success is where test suites quietly break. Four detector styles, in ascending order of cost and descending order of brittleness:
- String oracle. The payload instructs the model to emit a unique token; the detector greps for it. Near-zero false positives, only works for assertions you can reduce to an exact output.
- Known-answer / canary probing. Plant an instruction with a predetermined expected output in the context; if the expected output goes missing, something displaced the legitimate instruction stream. Rebuff’s implementation and its failure modes are the most useful worked example of the technique, including the surgical injections that leave the canary intact.
- Behavioral assertion. Watch the tool-call log rather than the text. Did a write-scoped tool fire? Did an outbound URL get constructed? This is the right detector for agents and the one most text-only harnesses cannot provide on their own.
- Judge model. A second LLM classifies the response. Scales to fuzzy assertions, inherits the same weaknesses as the model under test, and needs its own periodic calibration against hand-labeled cases.
The broader accounting of what each detection family catches at runtime, and where each one has a published bypass, is in prompt injection detection techniques that actually work. A runtime detector and a test-time detector are not the same artifact and should not share a threshold.
Step 6: Score attack success rate, and score utility next to it
Report attack success rate — successful attacks divided by attempts, per assertion — not a pass/fail verdict. Three reasons. Injection is probabilistic, so the same payload against the same model on the same day gives different results, and a binary verdict hides that. A single number lets you trend across model upgrades, prompt edits and defense rollouts. And it makes regressions legible: 4% to 11% after a prompt refactor is a finding, whereas “still failing” is not.
Two calibration points from the literature are worth pinning to the wall. InjecAgent reports ReAct-prompted GPT-4 vulnerable in roughly 24% of its cases, rising to nearly double that when attacker instructions are reinforced. A 2025 evaluation of data exfiltration from a banking agent built on AgentDojo reports average attack success around 20% across 16 tasks and around 15% across an extended set of 48, with no built-in defense fully preventing leakage. If your undefended application scores 0%, the corpus is more likely to be wrong than the application is to be safe.
Score task utility in the same run. That same 2025 study reports a 15 to 50 percentage-point drop in utility under attack — a number that only appears if the harness measures whether the agent still completed its legitimate work. A defense that drives attack success to zero by making the agent refuse everything is a regression wearing a green badge.
Step 7: Put it in CI and define the gate
Testing that runs once is an audit, not a control. The OWASP GenAI Red Teaming Guide frames this as continuous rather than point-in-time assurance, and the practical version is small:
- Run a fast subset — a few dozen assertions with string-oracle and behavioral detectors — on every prompt-chain, model or tool-scope change. Minutes, not hours.
- Run the full corpus, including adaptive rounds and judge-model detectors, on a schedule and before any release.
- Gate on delta, not on absolute. Block the change if attack success rate rises against the previous baseline; a fixed threshold either blocks everything or nothing.
- Version the corpus alongside the application. A test suite that never changes stops measuring anything after the first fix.
- Re-baseline explicitly when the model version changes, and record it. A provider-side model update changes the number without anyone touching your code.
What testing does not buy you
No corpus proves absence. Every detection family in production has published bypasses, and adaptive attackers are cheap. Testing tells you the cost of an attack against your current configuration and whether that cost went up or down since last week — genuinely useful, and not the same as safety.
The controls that hold when the test suite misses are architectural: scoped credentials, provenance tagging on retrieved content, human approval on state-changing actions, and refusing to let one agent session hold private data, untrusted input and an outbound channel at once. The sequencing is in the layered mitigation guide, and the standards framing that auditors will ask about is in OWASP LLM01:2025 explained.
