250 Seconds To Breach: What Anthropic Isn't Telling You
AI attacks just got (publicly) easier. Plus Avoiding AI Pitfalls: What to ask your would-be vendor | Edition 20
AI attacks just got (publicly) easier, plus avoiding AI pitfalls: What to ask your would-be vendor | Edition 20
Anthropic just dropped a truth that should scare anyone deploying AI: Not only is the attack surface effectively infinite, it’s also incredibly easy to hit. When it comes to large scale poisoning, Anthropic has experimentally determined that the cost of doing business could be as low as controlling 250 documents.
(Protip: Depending on the target, it might be lower.)
Everyone in-the-know in AI security already realized that poisoning attacks were virtually unmitigable, and weren’t going away. Now Anthropic has publicly quantified a little of what AI black hats already knew: the attack surface isn’t just enormous–it takes far less effort to breach than most AI security researchers realized.
This isn’t news to me, as I’ve been in talks about this from my research with colleagues at the OWASP AI Exchange for months. The exact same mathematical realities that make the attack surface near infinitely large, that make hallucinations a mathematical inevitability, also enable these types of attacks. There is no escaping it. As I’ve often said, it’s just math.
And math doesn’t care about your feelings.
What the Anthropic piece isn’t saying out loud: Not only is the attack surface shockingly easy to breach, it’s also shockingly easy to find.
Anthropic tested (or at least published) on the very narrow application of causing a model to produce gibberish text. If this is meant to obscure the real targets, or the ease with which they can be effectively hit, it’s doing a poor job.
Go ahead and crunch the numbers if you like–I already have: It would take even an amateur would-be attacker very few iterations to model out how to execute this type of poisoning attack with the goal of doing something that matters.
If they didn’t know how already. And that’s a big ‘if’.
And if I thought of it, please believe your system’s would-be attackers have too.
Counting Games
I’ve written before about how current AI Red Teaming ignores the mathematical reality of AI, and how, in its current state, red teaming will never get us to anything approaching quantifiable security.
I’ve personally called out AI Red Teams for bragging about how many test attacks they’ve supposedly deployed, without quantifying what that means.
Not one of them has been able to explain exactly what their product does beyond what appear to be spray-and-pray tactics at scale.
I’m on the lookout for the team that proves me wrong. Because these problems are not unknowable, nor are they intractable. And the first teams to get them right will have the decisive advantage in the future.
Now I want us to do a little thought experiment. In the age of GenAI, how long do you think it would take a determined attacker to generate 250 or so documents capable of this type of attack?
My money’s on seconds. At the moment, 250 seconds might seem like an overly optimistic estimate. But I’d say that view is naive.
From evasion to prompt injection to poisoning, attacks against AI are often treated as fully different classes when the same mathematical and engineering realities enable them all.
Just remember: If I can work out the math of how to evaluate adversarial potential in a way that actually works, so can your actual adversaries.
This research should be a wakeup call to the industry: If red teams were unaware of these techniques before, they’d better learn fast.
Because bad actors aren’t waiting.
Defining The Rules
It’s common knowledge among what the industry often terms “ethical hackers” and blackhats alike that there is one attack vector that always pays off.
It’s so powerful that it’s often banned in the contracts that govern professional security tests.
What is it? We’ll come back to that in a moment. But first I want to focus on the realities of what it actually means to test a security system. Not just to test, but to do it legally.
When you, as a theoretical red teamer, are hired to test a system, generally you don’t just show up and start trying to break things. These engagements are most often agreed on in advance–this includes everything from the cost to what systems can even be tested in the first place.
Before any hacking ever begins, all parties must agree on what’s being tested, and how. All of these agreements are codified in contracts prior to the start of work.
These contracts, called rules of engagement, provide the rules for exactly what systems and/or attacks are considered to be in scope. These rules may cover people or groups, attack types and methods, or systems and tools, as a few examples.
An ROE might contain instructions for testing certain employee groups–accounts of individuals or groups with certain sensitive or highly privileged access might be important to test, in certain use cases. Or certain accounts may be off limits–it’s easy to imagine a scenario in which a business might not want random pentesters reading their CEO’s email.
Another example: a typical pentest engagement might involve investigating IP addresses. But a pentester who performs certain actions on IP addresses which are not in scope could be breaking the law, despite their best intentions. Ending up in legal hot water is something no security professional wants, so it’s important to get this right. A clearly spelled out ROE helps with this.
When you deploy an AI system, there’s no ROE. It’s the model, the interface, and whatever guardrails versus the user.
And for users, whether friendly or foe, anything goes.
The Most Powerful Tool In Hacking
What technique is so effective it’s often banned in ROEs?
It’s not a software tool, a framework, or some special flavor of Linux.
It’s social engineering.
Social engineering is often treated like a separate field or technique, but in reality, the principles that make it work underscore every effective strategy in offensive security. Social engineering works by using established (social) systems and norms for malicious purposes.
It uses the systems humans trust to keep them safe against them. Isn’t that really the heart of hacking?
There are many potential reasons why this type of tactic can be so effective; I’m not here to argue about which might be more or less valid. But I would submit that if we zoom out and take a systems-level approach, what emerges is a commonality among many hard-to-defend vulnerabilities: the attack surface is simply unquantifiably massive.
This is why phishing training never stops (or becomes effective). It’s because you can never foresee or train against all the potential attack vectors.
Your intuition will confirm it for you: There are a near-infinite number of ways of using natural language to talk someone into doing something they shouldn’t.
Training against these attacks has limited value. Attackers will improve their techniques, and people will always fall for something another human wouldn’t expect. There’s no silver bullet method to teach people, and there isn’t a patch.
In other words, the attack surface is effectively infinite. And thus, to a degree, undefendable.
Can we help humans improve their odds of beating social engineering attacks? Absolutely–in large part because humans have neuroplasticity and the ability to truly learn and absorb context.
AI does not. And in this respect, the human factor is easier to defend than the machine.
AI’s Rules of Engagement: Single Channel, Infinite Attack Surface
GenAI systems that rely on natural language prompts have, built into them, what you can think of as the single channel problem: data and instructions flow through a single channel, i.e. a prompt, making it impossible to fully separate the two–and enabling the passing of potentially malicious instructions to the system.
This architectural reality combines with the mathematical realities of statistically based learning and natural language to create the vulnerabilities that plague these systems.
Many people have likened testing for GenAI’s vulnerabilities to social engineering. I disagree with that comparison strongly on both engineering and epistemological grounds.
But for now, I’d like for you to focus on what the social engineering and AI attack surfaces have in common: their sheer, unmitigable size.
There are nearly infinite ways to trick people into doing things–and this is tied directly to the combinatorics of natural language and its associated contexts and meanings. Similarly, natural language provides nearly infinite ways to interact with–and maliciously manipulate–GenAI/Agentic systems.
The categories of attacks that security researchers have discovered in GenAI systems represent an already massive swath of potential attack vectors. It’s a goldmine for researchers–and whatever the opposite of a goldmine is for organizations deploying, and ultimately defending them.
The Threat Model
Poisoning attacks against GenAI are easier and cheaper than most practitioners imagined, but how to defend against them remains challenging.
Social engineering and AI security testing aren’t the same thing, but they do share a commonality: an equally massive attack surface.
Humans have a stateful mind, neuroplasticity and context-awareness; AI does not–so in a sense even social engineering is easier to defend against than AI attacks.
Resources To Go Deeper
It’s been pointed out that work like Anthropic’s recent research may have implications for in-context learning. So I’m providing foundational resources here for a deeper understanding.
Zhao, Shuai, Meihuizi Jia, Anh Tuan Luu, Fengjun Pan and Jinming Wen. “Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning.” Conference on Empirical Methods in Natural Language Processing (2024).
He, Pengfei, Han Xu, Yue Xing, Hui Liu, Makoto Yamada and Jiliang Tang. “Data Poisoning for In-context Learning.” ArXiv abs/2402.02160 (2024): n. Pag.
Qiang, Yao, Xiangyu Zhou and Dongxiao Zhu. “Hijacking Large Language Models via Adversarial In-Context Learning.” ArXiv abs/2311.09948 (2023): n. Pag.
Executive Analysis, Research, & Talking Points
Five AI Pitfalls To Avoid: What To Ask Your Would-Be Vendor
The sometimes intractable, often mission-critical vulnerabilities in PredAI systems are really only just becoming widely known in industry.
Deploying these systems successfully quite literally requires deploying them securely. And secure AI deployment requires expertise.
These five things business leaders need to look out now for in the current AI landscape represent the potential for tremendous business loss, if not handled properly. Since nobody is talking about this, and the market is incentivized to keep decision makers in the dark, I’m giving them here–along with questions that can help root out the real engineering truth from marketing gloss.
