The Attacker Moves Second:
Why AI Systems Break So Easily

The unsolved problem at the centre

Every LLM application shares one structural weakness: the model cannot reliably tell the difference between instructions from its developer and instructions hidden in the data it is asked to process. This is prompt injection, and it sits at position one — LLM01 — on the OWASP Top 10 for LLM Applications, the field's reference taxonomy [1].

It is not a bug awaiting a patch. In late 2025 Meta's AI security team stated plainly that "prompt injection remains an unsolved problem, and attempts to block or filter them have not proven reliable enough to depend on" [2]. The most striking evidence came from a joint paper by researchers at OpenAI, Anthropic and Google DeepMind: when they allowed adaptive attackers to move second against twelve recently published defences, attack success rates exceeded 90% for most of them — even though those same defences had originally reported near-zero success. A human red-teaming competition reached 100% [3].

Indirect injection: the attacker never talks to you

The dangerous version of prompt injection is indirect. The attacker does not type into your chatbot; they plant instructions in content the system will later read — a web page, an email, a document, a code comment, a support ticket. The foundational academic work on this, Greshake et al.'s 2023 paper aptly titled "Not what you've signed up for," demonstrated exactly this against real, LLM-integrated applications [5]. The more tools and data sources an agent can reach, the more surfaces an attacker has to write on.

EchoLeak: the theory became a CVE

For a long time these attacks lived in research papers. In 2025 one became a catalogued vulnerability. EchoLeak (CVE-2025-32711) was a zero-click flaw in Microsoft 365 Copilot: a single crafted email, never opened or clicked, could cause Copilot to leak data from the user's own documents and messages by chaining an indirect prompt injection with an automatic content-rendering bypass [6]. It was, as the discovering researchers put it, the first known zero-click attack on an AI agent — and it required no action from the victim at all.

The exfiltration mechanism is worth understanding because it recurs: an LLM is induced to emit a markdown image link pointing at an attacker's server, with stolen data encoded in the URL. The victim's client dutifully fetches the image — and the data is gone. No malware, no download, just the system's own helpfulness turned against it.

A guardrail vendor advertising 95% effectiveness is describing a failure rate. In security, a control an attacker beats one time in twenty is a control an attacker beats.

Where the real boundary is: agency

What turns an injection from embarrassing to catastrophic is what the compromised system can do. The OWASP list calls this Excessive Agency (LLM06); the security researcher Simon Willison calls the danger the "lethal trifecta" — an agent that simultaneously has access to private data, exposure to untrusted content, and the ability to communicate externally [7]. Any system holding all three can be steered into exfiltrating what it knows. Remove any one leg and the attack loses its exit.

Retrieval-augmented systems inherit their own version. In PoisonedRAG, researchers showed that injecting as few as five malicious documents into a knowledge base of millions was enough to control a system's answer to a targeted question 90% of the time [4]. If anyone can write to your knowledge source — a wiki, a ticketing system, an indexed shared drive — they can, in principle, write to your model's conclusions.

Building as if the attacker moves second

There is no single defence, because there is no reliable filter. What works is architecture that assumes injection will sometimes succeed and limits the damage when it does:

• Constrain agency. Design tools so a hijacked agent cannot take consequential actions without a human in the loop. Break the lethal trifecta by design.
• Treat all retrieved content as untrusted — including your own documents, which an insider or an attacker may have edited.
• Control egress. If the model cannot reach arbitrary external destinations, it cannot exfiltrate to them.
• Red-team adaptively. Static test suites certify nothing; the only meaningful test is an attacker allowed to adapt, as the 2025 research made unmistakable.

The uncomfortable truth is that LLM applications are, today, easier to attack than to defend. That is not a reason to avoid building them — it is a reason to build them the way we build other systems that face determined adversaries: assuming the first line will be breached, and engineering so that the breach is contained.