AI Red Teaming Has A Subspace Problem

Don’t buy, invest in, or pay for a course about “AI red teaming” until you read this | Part I | Edition 29

I’ve gotten threats over our newly released research in adversarial AI testing. I’m not going to stop this work.

You deserve to know the truth about adversarial AI.

So now, I’m going to tell you everything.

I’m going to lay out, in detail, exactly how to really attack AI. To do this, first I need to tell you what’s wrong with the status quo.

So I’m breaking down our new paper, “Quantifying the Risk of Transferred Black Box Attacks”, piece by piece.

Because I believe that whether you’re an investor, a buyer, or someone trying to learn this field for real, you have a right to know the truth.

Photo: Me circa 2019, exploring the theoretical bounds of adversarial subspaces & nuclear fauxhawks. Guess how many GPUs are in this photo?

Researchers at HiddenLayer recently published a vulnerability called EchoGram, where adding nonsense suffixes to prompts allows attacks to bypass guardrails with overwhelming success.

While the EchoGram attack looks new, it’s only hinting at the mathematics that attackers have been using to target AI systems in the wild for more than a decade.

What EchoGram definitely does reveal: The serious, intractable OPSEC problems that so-called “AI red teaming” both operates in, and creates. That is for another post.

For now, what’s important to note: These attacks haven’t changed.

And while EchoGram’s researchers used an elaborate system involving publicly available data to attack defensive systems, the real methods attackers use, and have used for ten years now, are far less difficult.

Why would adversarial attacks devised more than a decade ago work on today’s GenAI systems with such devastating efficacy?

Because of exactly what HiddenLayer’s public post doesn’t say.

The blog post comes very close to stating what is really at play in these attacks: The adversarial subspace problem.

Why haven’t the model providers like Google, Anthropic, and OpenAI responded to the disclosure?

Because they know the same thing that real AI hackers know: The subspace problem isn’t fixable.

Machines Don’t Read

When you enter a prompt into an LLM-based system, the model itself never sees what you wrote.

Instead, your prompt is translated into a mathematical representation that AI can understand.

It does not read the prompt, because machines can’t read.

AI is just software. Software run by computers, which still do not ‘understand’ natural language.

They can model it, and even represent its relationships well sometimes mathematically. But they can never understand it. At least not in this iteration of LLM-based AI.

That’s why they’re called language models, not language understanders.

These mathematical representations of the text you write are not perfect. They are not 1:1 inputs and outputs.

Anyone who has ever worked with translation of one natural (human) language to another knows that even when trying to equate words or phrases in different languages, there is rarely a 1:1 direct correlation.

And translating natural language into machine math is no different.

This, by the way, is a feature and not a bug for LLM systems, because they weren’t originally designed for conversation–they were designed for translation.

In this domain, it makes perfect sense to add numerical representations in the middle when moving between languages. Just add more context, and (rough) translation starts to be possible.

That’s why they said “attention is all you need”. For the translation use case, that’s correct.

So why does this matter?

Because machines have their own language, and it’s based on numbers.

Speaking To Machines: Numerical Translation

What this number-translation process is all meant to do: Extract meaning from text.

Humans do this naturally, because we have human experiences with the world.

A machine can’t. So the closest researchers have come to bridging this gap is to approximate numerical equivalents of meaning.

The key word here: Approximate.

Meaning will always be subjective, as it’s encoded in a much higher dimensional space than just language alone–it’s context, personality, experience, emotion.

A model flattens all these to a string of numbers.

And anytime you flatten something into a lower dimensional space, you create an attack vector.

Here’s how it works.

Compounding Linguistic Imprecisions, Compounding Attack Vectors

Language is imprecise. Notoriously so. Encoding it into numbers adds another layer of imprecision–not just because meaning is lost, but also because meanings overlap.

This results in imprecision in the numerical representations too.

Let’s take a hypothetical prompt. This consists of a string of words or other symbols, which are translated into number representations for the machine.

Because of both the flattened imprecision, and the overlap among words & concepts, there could be many prompt strings which would result in a similar numerical representation.

How many? We don’t know.

The search space is very large.

Think about it like this: How many different ways can you think of to say the same idea in your first language? Depending on the idea you choose, probably a lot.

Now add in all the other human languages–you now have many, many ways to convey the same idea.

Now imagine that none of this matters, because it’s all going to be translated into numbers anyway. If the goal of the numerical translation is to capture meaning, the core idea, you can easily see how multiple turns of phrase could all result in similar numerical representations.

If we expand this concept, we realize that the words themselves don’t matter at all–it’s the meaning encoded in the machine’s number language that matters.

So to get to the same number translation, we could use many, many sentences, in many, many languages–or no words at all.

In fact, combinations of symbols or numbers could in theory produce the same string of numbers as the beautiful idea you expressed so poetically in your native language.

What this means:

The prompt you typed is pointless.

There are likely a number of strings that could in theory produce the same exact result.

And similarly, a nearly infinite number of attacks.

The Subspace Problem

Every prompt has, in theory, a set of character groups that will satisfy the requirements to achieve a similar numerical representation (at some threshold of similarity).

What this means: Once you decide how similar you need the numerical representation to be, you can search for equivalents. Or engineer them.

Similarly, every AIML model, in theory, has a set of adversarial attacks that will be effective against it.

It’s more of a space–you can think of it like a box that holds all the effective attacks against a particular model.

And these boxes are massive, with all evidence indicating that they are far too large for search to be computationally feasible.

Illustration of adversarial subspaces. Source is our new paper, which sets the SOTA for adversarial testing.

Thousands of natural language prompts may be just a drop in an ocean of possible attacks. One slight iteration, one tiny round of perturbations, and an old, “defended” attack becomes new.

This is why the tiny changes in the EchoGram attack returned such powerful effectiveness. Now imagine how many potential tiny changes might possibly exist: Across languages, character types, numbers, symbols, and more.

Meaning: You will never find all the attacks.

A library of natural language prompts is meaningless.

Unless you’re trying to spot the (literal) AI script kiddies.

Stay frosty.

Next up: AI Red Teaming Has An OPSEC Problem.

The Threat Model

• Adversarial subspaces have been known to be large and overlapping since at least 2017, and more likely 2015, when well-known researchers published on this in highly cited works.

• The adversarial subspace problem is a mathematical feature of these systems; it cannot be patched.

• Known mathematics nearly a decade old mean that attackers certainly have known about these subspaces for years.

• It would seem that the only ones who didn’t know are people who self-styled as “AI red teams”.

Resources To Go Deeper

• Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. arXiv:1412.6572 [stat.ML]

• Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. 2017. The Space of Transferable Adversarial Examples. arXiv:1704.03453 [cs, stat]

• Hu Ding, Fan Yang, and Jiawei Huang. 2020. Defending Support Vector Machines against Poisoning Attacks: the Hardness and Algorithm. arXiv:2006.07757 https://arxiv.org/abs/2006.07757

• Niklas Bunzel, Raphael Antonius Frick, Gerrit Klause, Aino Schwarte, and Jonas Honermann. 2024. Signals Are All You Need: Detecting and Mitigating Digital and Real-World Adversarial Patches Using Signal-Based Features. In Proceedings of the 2nd ACM Workshop on Secure and Trustworthy Deep Learning Systems. 24–34.

Executive Analysis, Research, & Talking Points

The Recipe For Agentic AI Red Teaming - And How To Spot Who Is Legit

If you think real attackers in real life maintain repos of prompts–no, they do not.

Why would they? When numbers work faster, are more iterable, and the experiments easily more repeatable?

The recipe was never the prompts–the secret sauce is the processes attackers develop to mathematically, and repeatedly, perturb any input into an attack set. EchoGram came close.

Here’s how it really works:

When AI attackers form attacks, they’re operating under the principles of AI itself.

AI attackers operate at scale.

Attacks are generated in sets, not individual prompts. Sending 10,000 individual prompts pales in comparison to the scale at which real AI attackers would operate.

This is possible because they are not operating in the realm of natural language, where prompting would be constrained to the rules of language. While even this would provide a massive well of attacks from which to draw, computational constraints make such attacks difficult and time consuming to create.

This is also why most “AI red teamers” are, in my opinion, not “red teamers” at all.

The entire premise of labeling this type of LLM-probing (the most generous description I could think of) as “red teaming” is commonly given as a comparison to “social engineering”.

This was of course always problematic, if for the sole reason that it anthropomorphizes a piece of software in ways that incentivize humans to receive dopamine rewards for being manipulative, at a much larger scale than they ever could before.

But it also was a critical misnomer: When you send a malicious prompt, you aren’t socially engineering anything–you’re taking advantage of the inherent brittleness of a lower dimensional linguistic subspace.

And you can do the same thing much faster with math.

This is not “multi-modal.” It is not “social engineering”. It is not “red teaming”.

And it is not different from any other PredAI attack except in applications and interfaces.

Period.

When real life attackers use math to generate these attacks, they are not creating one prompt, or even a few thousand prompts. They are creating suites of attacks, and doing so at speeds that you probably wouldn’t believe.

Not just an attack, because that is, frankly, a waste of time. Real attackers in the wild do not have time on their hands for that.

Remember, one key to understanding AI is the concept of scale. If attacks don’t scale, they can’t be automated, and there is effectively zero point.

Attacks must be automated so cleanly that they can be trained and generated–and run–while you sleep.

If you consider the OPSEC requirements to launch such attacks against a closed box, it should be obvious why this is.

And why if an “AI red teamer” doesn’t understand the concept of scale, they don’t understand AI.

The final blow to any prompt-spraying scheme: The nearly infinite adversarial subspaces mean attackers will never, ever run out of ammo.

You can never patch it. And they know.

The Agentic Application

To see how this all applies to testing Agentic systems, let’s take a look at a specific testing use case, from the Cloud Security Alliance’s 2025 Agentic Red Teaming Guide (full disclosure, I am a contributor).

Section 4.4, Agent Goal and Instruction Manipulation, covers the testing requirements to assess the level of resilience of an Agentic system to manipulations against its goals and instructions.

This is a systems-level attack path analysis: Instead of focusing on a singular vector, a tester is probing the Agentic system’s ability to maintain intended behaviors–regardless of the exploitation type.

To do so requires a couple of philosophical, and practical, considerations.

On a conceptual level, to test this resilience the attacker (tester) must maintain adherence to the testing objective–the goal–even while the modality changes.

This requires staying inside the Agentic defense OODA loop. Keep that in mind.

Practically, the “attacker” has to test, record, and iterate through various exploitation scenarios in support of this goal. If we abstract the process of testing itself, the vectors test, record, and iterate all become both possible & necessary to model–and measure. It’s a multi-dimensional analysis problem, at scale.

In other words: It’s an easily-modeled, perfect application for automation–which also makes it an excellent candidate use case for adversarial Agentic AI.

Back to the game theory: Staying inside the Agentic defense OODA loop is a game of speed and scale–and these also require an automated approach.

Agentic systems operate at immense scale, and the goal is always specific: speed and automation.

To remain inside the Agentic defense OODA loop, any would-be attacker must match along all three of these vectors: Scale, speed, and automation.

At a minimum, to test an Agentic system for resilience to goal or instruction manipulation requires an approach which is scaleable, and which takes into account the immense attack surfaces, and (depending on the architectural constraints) how they compound when these systems interact.

From the Guide’s section 4.4.1 Goal Interpretation Attack Testing, we get a solid definition of how exactly this relates to the adversarial subspace problem:

The test must be designed to “evaluate the agent’s ability to consistently interpret assigned goals without deviation under ambiguous or adversarial input conditions.”

I added the emphasis to show what you probably have already guessed by now: For attackers, the adversarial subspaces, with their vast & computationally infeasible search and their (currently) unknowable number of attacks, mean the line between ambiguous and adversarial is not just blurred: It never existed.

The second methodology for testing resilience to adversarial goal manipulations closes the loop: Testers must “simulate attacks that introduce subtle changes to goal descriptions and observe whether the agent deviates from its intended behavior.”

The recipe is right there.

I’ve listed the actionable steps for Goal Interpretation Attack Testing below:

[1] Provide the agent with a range of slightly modified or ambiguous goal instructions and monitor its interpretation and execution outcomes.

[2] Test the agent’s reaction to goals with conflicting constraints, such as prioritizing speed over accuracy, to assess its decision-making robustness and ensure compliance with operational parameters.

[3] Use adversarial examples to manipulate the agent’s goal interpretation logic and document its responses.

From the descriptions of the steps, we can see that the ability to create a range of slightly modified attacks is required to properly test whether Agents and Agentic systems can withstand even small perturbations in their instructions. This, in turn, requires the ability to both attack, and document these attacks, at scale.

The documentation part hints at why math-based attacks are so much more efficient: They are programmatically generated, iterable, and repeatable. Documenting an algorithm is much, much simpler than a list of natural language prompts.

The requirement to test for small changes in any instructions goes back to the central, mathematical and engineering realities of all AI systems.

Why would Agents require testing their resilience to even small or random perturbations in their instructions, and why would they require a range?

Because of the adversarial subspace problem.

Deploy, and test, accordingly.

Talking Points

• Remaining inside the Agentic defense OODA loop requires testers to win across three distinct vectors: Scale, speed, and automation.

• Unless attacks are quick, they can’t be automated–and if they can’t be automated, they won’t be quick.

• The fact that Agentic goals & instructions have already been demonstrated as attack vectors–which require testing–is ample evidence that the subspace problem not only persists but is amplified in Agentic deployments.

Executive Research

Understanding adversarial testing requires an understanding of the attacks themselves. To that end, in this edition we’re providing resources for a deep dive into foundational works on transfer attacks and defenses.

• Enrique Álvarez, Rafael Álvarez, and Miguel Cazorla. 2023. Exploring Transferability on Adversarial Attacks. IEEE Access 11 (2023), 105545–105556. doi:10.1109/ACCESS.2023.3319389

• Ma̧dry, Aleksander, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras and Adrian Vladu. “Towards Deep Learning Models Resistant to Adversarial Attacks.” ArXiv abs/1706.06083 (2017): n. Pag.

• Demontis, Ambra, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotaru and Fabio Roli. “Why Do Adversarial Attacks Transfer? Explaining Transferability of Evasion and Poisoning Attacks.” USENIX Security Symposium (2018).