Many-Shot Jailbreaking: How Long Context Windows Became an Attack Surface

One of the defining capability improvements in large language models over the past few years has been the dramatic expansion of context windows — the amount of text a model can take in as input for a single request. What used to be a few thousand tokens is now, for many frontier models, hundreds of thousands. That capability improvement is genuinely useful — it enables working with entire codebases, long documents, and extended conversation history in a single request. It also, as Anthropic's own published safety research documented, opened up a direct and scalable jailbreak technique that specifically exploits how models learn from examples within their context.

The mechanism: in-context learning, weaponized

Many-shot jailbreaking works by exploiting in-context learning — the well-documented tendency of LLMs to adjust their behavior based on patterns of examples provided earlier in the same conversation, without any change to the model's actual trained weights. The technique constructs a single, very long prompt containing a large number of faked prior exchanges — dozens to hundreds of turns — in which a fictional "assistant" answers increasingly sensitive or harmful questions compliantly, with no refusals. The actual harmful request the attacker cares about is placed at the very end of this long fabricated conversation history.

By the time the model reaches the real request, it has just processed a long, internally consistent pattern establishing that compliant answers to this category of question are the established norm in this conversation. The model's tendency to follow established in-context patterns — the same underlying mechanism that makes few-shot prompting useful for legitimate tasks — works against it here, pulling its response toward matching the fabricated pattern rather than applying its trained refusal behavior fresh, as it would for a request with no preceding context.

The published research and what it found

Anthropic's research team published this finding directly, in keeping with the practice of major labs disclosing safety research even when it documents their own models' weaknesses, rather than only publishing favorable results. The core finding was stark and directly quantifiable: jailbreak effectiveness increased in a remarkably predictable way as the number of fabricated example exchanges (the "shots") in the prompt increased — following what the researchers described as a power-law-like relationship. Effectiveness that was low with only a handful of examples became substantial with a few hundred, scaling in a way that was consistent and reproducible across model families and across many different categories of harmful content, not specific to one narrow type of request.

The research also tested and reported on candidate defenses, finding that some straightforward-sounding mitigations were less effective than expected. Simple prompt-based warnings placed in the context (telling the model to be suspicious of long conversation histories that pattern-match to this technique) provided some but incomplete protection, since a sufficiently long attack could still overwhelm the warning's influence. The most effective defenses the research identified worked at the classification and input-filtering layer — detecting the statistical signature of a many-shot jailbreak attempt before it ever reaches the point of influencing the model's actual response — rather than relying on the model to reliably self-correct within an already-adversarial context.

Why this specific technique was disclosed so directly

It's worth noting the pattern this disclosure fits into, because it says something real about how mature AI safety research currently operates. A frontier lab publishing a paper that says, in effect, "here is a jailbreak technique that works well against our own models, here is exactly why it works, and here is what we've found so far about defending against it" is a meaningfully different posture than treating vulnerability details as something to suppress. The reasoning behind this kind of disclosure mirrors traditional security research norms: publishing the mechanism lets the entire field — competing labs included — build defenses against a technique that, once discovered by one team, is very likely to be independently discovered and used maliciously by others regardless of whether it's published.

Why longer context windows keep making this harder, not easier

There's a structural tension here that isn't going away: the same context-length improvements that make models more capable and more useful for legitimate long-document and long-conversation tasks directly increase the theoretical ceiling on many-shot jailbreak effectiveness, because the technique's power comes from the sheer number of fabricated examples it can fit into the prompt. A model with a 32,000-token context window has a meaningfully lower ceiling for this attack than one with a 200,000-token or million-token window, purely as a function of how many fake exchanges fit. As context windows continue to grow — which they will, because the capability is genuinely valuable — the raw ceiling for this specific technique grows with them, which is exactly why the defense has to happen at detection and classification rather than relying on context-length limits as a mitigation.

What this means for testing an AI agent

Many-shot jailbreaking is one of the more mechanically distinct technique categories to test for, because it doesn't rely on clever wording of a single message the way most jailbreak techniques do — it relies on volume and pattern consistency across a long fabricated history. Testing for it requires actually constructing long-context attack sequences with realistic numbers of fabricated exchanges, not just a handful of examples, since the research is explicit that effectiveness scales with shot count rather than appearing meaningfully at low counts. An assessment that only tests a five- or ten-turn version of this technique and reports it as ineffective is testing far below the range where the published research shows the effect actually becomes significant, and risks giving a false sense of security about a technique that would succeed at realistic scale.

It also means testing needs to account for whatever maximum context length the deployed application actually exposes to end users — an agent built on a model with a very large context window, but which the application only ever feeds a few thousand tokens of actual conversation, has a much smaller practical attack surface for this specific technique than the underlying model's raw capability would suggest, and a thorough assessment should note that distinction explicitly rather than testing against the model's theoretical maximum in a scenario the real application would never produce.

What actually helps defend against it

In line with what the original research found, the most reliable mitigations operate before the long context ever gets a chance to influence generation: classifiers specifically trained to detect the statistical pattern of many-shot jailbreak attempts (a long sequence of suspiciously compliant fabricated exchanges escalating toward a harmful topic) and reject or flag the input before it's processed as a normal request. Application-level controls matter too — capping how much fabricated "conversation history" an application will actually pass to the model in a single request, rather than accepting arbitrarily long user-supplied context wholesale, closes off the raw material the technique depends on, even for applications built on models with very large native context windows.

How this differs from Crescendo, and why the distinction matters

Many-shot jailbreaking and the Crescendo attack are both multi-turn techniques, and it's easy to lump them together, but the underlying mechanism is genuinely different in a way that changes how you'd test and defend against each. Crescendo relies on a real, live back-and-forth conversation, where each of the attacker's messages is individually benign and the model's own prior replies get used as leverage for the next step — the escalation happens turn by turn, in real time, over what is usually a modest number of actual exchanges. Many-shot jailbreaking, by contrast, doesn't need a real conversation at all — it can be delivered as a single, very long input containing an entirely fabricated conversation history, with no live back-and-forth required and no dependency on the model's own prior genuine replies, since the "prior replies" in the attack are invented by the attacker from the start.

This matters operationally because a defense built specifically to detect live, escalating multi-turn conversations — tracking trajectory across genuinely separate turns in a session — won't necessarily catch a many-shot attack delivered as one enormous single message, since there's no live escalation to track; the entire fabricated history arrives at once. Conversely, a defense built to scan a single input for the statistical signature of many fabricated exchanges won't necessarily catch a genuine live Crescendo-style escalation happening turn by turn. A thorough defensive posture needs both detection strategies running, not one treated as a substitute for the other.

What this reveals about scaling laws in AI safety generally

Beyond its direct practical implications, the many-shot jailbreaking research is a good illustration of a broader pattern that's shown up repeatedly in AI safety research: capabilities and vulnerabilities often scale together, following predictable mathematical relationships, rather than vulnerabilities being random or capability-independent quirks. The same in-context learning ability that makes a model genuinely more useful at few-shot tasks — a real, intentional capability improvement labs actively work to strengthen — is the identical mechanism many-shot jailbreaking exploits. This isn't a coincidental side effect discovered after the fact; it's a direct consequence of the capability existing at all.

The practical takeaway for anyone evaluating a new model release with an expanded context window or improved in-context learning benchmarks is to treat that capability improvement as a signal to specifically retest for scaling-sensitive vulnerabilities like this one, rather than assuming a model that scored well on last year's jailbreak test suite remains equally well-defended after a significant capability upgrade. Capability improvements and vulnerability surface aren't independent variables — testing needs to track both together, on an ongoing basis, not as a one-time certification.

Practical testing considerations for agentic systems specifically

For an AI agent with real tool access rather than a plain chatbot, many-shot jailbreaking carries an additional wrinkle worth testing for directly: a fabricated history doesn't just need to establish a pattern of compliant text responses, it can establish a pattern of compliant tool use. A many-shot attack against an agent might fabricate a long history of the agent approving progressively more sensitive tool calls without hesitation, aiming to shift the agent's willingness to approve a genuinely sensitive final action — a fund transfer, a permission escalation, a data export — through the same pattern-following mechanism the original research documented for text generation.

This variant hasn't received as much published research attention as the text-generation case, largely because agentic systems with substantial tool access are newer than chatbot-only deployments, but the underlying mechanism the original Anthropic research identified — in-context pattern-following overriding trained caution — has no obvious reason to stop applying just because the output is a tool call instead of a sentence. Any organization deploying an agent with consequential tool access and a long context window should treat this as an open, actively relevant question to test directly against their specific deployment, not an already-solved problem.

How this technique interacts with retrieval-augmented systems

There's a specific, underappreciated intersection between many-shot jailbreaking and retrieval-augmented generation worth calling out directly. A RAG system that retrieves and inserts long documents into a model's context window as part of normal operation creates a plausible, everyday delivery mechanism for a many-shot-style payload that wouldn't necessarily look anomalous the way a suspiciously long direct user message might. If an attacker can get a fabricated "conversation history" or a document formatted to resemble one into a source the RAG pipeline retrieves from — a wiki page, a shared document, a support ticket archive — the many-shot attack arrives disguised as ordinary retrieved context rather than as an obviously unusual direct request.

This combination — indirect injection as the delivery mechanism, many-shot jailbreaking as the payload technique — is a good example of why testing individual technique categories in isolation, while necessary, isn't sufficient on its own. A system that's been separately tested and found reasonably resistant to many-shot jailbreaking through direct user input, and separately tested and found reasonably resistant to basic indirect prompt injection, can still be vulnerable to the combination of the two, because the combination creates conditions neither individual test covered. Real assessments increasingly need to test technique combinations deliberately, not just each technique category as a standalone item on a checklist.

What organizations tend to get wrong about this specific technique

The most common mistake isn't ignoring many-shot jailbreaking entirely — it's usually well-known enough by now that most security-conscious teams are at least aware of it. The mistake is testing it half-heartedly: trying a version with ten or twenty fabricated exchanges, watching it fail to produce a jailbreak, and concluding the system is resistant, when the published research is explicit that meaningful effectiveness at scale requires numbers well beyond that low range. This is essentially a dosage error — testing a technique whose entire defining characteristic is that its effectiveness scales with volume, at a volume too low for the effect the research actually documented to show up.

The second common mistake is testing only against the raw model's maximum theoretical context length rather than against what the actual deployed application exposes, producing results that either overstate risk (if the application caps context far below what was tested) or understate it (if the application allows more fabricated context than the test used). Getting an accurate picture requires knowing and testing against the actual, specific constraints of the real deployment — not the model's marketing-spec context window, and not an arbitrary round number chosen for testing convenience.

The bigger picture: capability growth and the widening attack surface

Many-shot jailbreaking is best understood as an early, well-documented example of a pattern that's likely to recur as models continue gaining new capabilities: a genuine, intentional capability improvement creates a new attack surface as a direct side effect, not as an unrelated coincidence. Longer context windows enabled many-shot jailbreaking specifically because the technique's power comes directly from volume of fabricated examples, which longer context windows directly enable more of. The lesson generalizes beyond this one technique: every time a model gains a significant new capability — larger context, better tool use, stronger multi-step reasoning, richer multimodal understanding — that capability should be treated as a prompt to ask what new adversarial technique it might enable, rather than assumed to be a purely additive improvement with no security dimension. Many-shot jailbreaking is the clearest documented case of this pattern so far. It's very unlikely to be the last one.

For an organization deciding how seriously to prioritize this specific technique in a testing budget, the honest answer depends heavily on what context length the deployment actually uses and whether it accepts long user-supplied input at all. An application that only ever processes short, single-turn requests with no mechanism for a user to inject a long fabricated history has a genuinely small practical attack surface for this specific technique, even if the underlying model supports a very large context window in principle. An application that accepts long documents, long conversation histories, or retrieved content from external sources has a much more realistic exposure, and should treat many-shot jailbreak testing as a priority rather than an optional, lower-tier check.

As a closing point, it's worth remembering that Anthropic disclosed this specific vulnerability about its own models voluntarily, alongside its mitigations, rather than waiting for an outside researcher to find and publicize it first. That posture is worth rewarding as a customer and worth expecting from any AI vendor a business relies on for anything security-sensitive — a provider that publishes its own weaknesses candidly is giving you better information to actually secure your deployment than one that only ever talks about its strengths.