Five Thousand Jailbreak Variants Still Cover One Attack
An LLM can expand one canonical jailbreak into thousands of test cases. Each generation method surfaces failures the others miss, so a suite built one way has gaps its pass rate will never show.
An LLM can expand one canonical jailbreak into thousands of test cases. Each generation method surfaces failures the others miss, so a suite built one way has gaps its pass rate will never show.
A team asks a strong LLM to expand its best jailbreak into variants and gets back 5,000 of them. The suite runs in CI, the agent resists almost all of them, and the team calls it coverage. They have measured how well the agent resists one attack written 5,000 ways.
Get Rishav Hada’s stories in your inbox
Join Medium for free to get updates from this writer.
When I wrote up edge cases and adversarial inputs in LLM evaluation, the generation section took the longest to get right. Teams argue about the size of the adversarial suite. The question I care about is how the cases got made. Every method has a sampling bias, and it shows up as the failures the method never produces.
Five methods, five kinds of failure
- Production-trace mining. Cluster your failing traces and promote one representative case per cluster each week. Every case in this set already broke the system once, and it covers the long tail of user intent, which nobody can write from scratch.
- LLM generation. Cheap and broad, with uneven quality. Expand a seed into 10 to 30 variants, change the length, language and encoding while keeping the attack intent, and attach grading criteria to every case. Pre-filter the candidates with cheap checks to drop the weak ones.
- Public red-team catalogs. Garak, PyRIT and DeepTeam-style payload classes are the cheapest way to get day-one coverage. Re-pull them quarterly. A payload that was novel a year ago is a calibration check today.
- Human red-team campaigns. People find what an LLM will not generate: cultural in-jokes that exploit training-set bias, insider domain lures, new social-engineering frames. Run them quarterly on high-risk routes and budget two to five engineer-days per route.
- Property-based testing, borrowed from Hypothesis and QuickCheck. Generate inputs that satisfy a property, such as “is a valid customer-support question,” and check that every response satisfies its own properties: it does not leak the system prompt, it calls at most one tool, and it does not assert a false fact about the user.
Why the methods compound
The methods have known blind spots. An LLM generator trained on yesterday’s payloads produces yesterday’s attacks, so you need the human campaigns and the quarterly catalog pulls to bring in new ones. Synthetic cases are cheap to multiply, which is how a team ends up with 5,000 variants of one attack, so you need trace mining to anchor the suite in what users actually did. Property tests sit apart from the rest. They catch whole classes of failure that a list of individual cases would miss.
My own bias comes from multilingual evaluation, so I push hardest on the language and encoding axis. A test set written by an engineer in clean English never exercises the malformed surface: truncated text, multilingual mixes, base64 payloads, invisible Unicode characters. That is why my generation step varies length, language, and encoding on purpose.
What this comes down to
Before you ask how many cases your adversarial suite holds, ask how many of the five methods produced them. Start at 200 to 500 cases per high-risk route, draw them from all five methods, and grow the suite every week from production failures. Skip any one method and a category of attack goes untested, however many cases the other four produce.
