Matic: A new era of visually intelligent cleaning (Sponsored)
Try Matic, backed by their 6 month money back guarantee.
In June 2025, security researchers at Aim Security demonstrated that a single email could cause Microsoft 365 Copilot to retrieve a company’s internal files and transmit them to an external server, with no user interaction.
The email contained ordinary text and exploited no software flaw. Copilot’s output was driven by hidden instructions in the email together with the user’s actual request, and the system applied no distinction between the two. Microsoft closed the specific path within days and reported that customers were unaffected.
But this incident sheds interesting light on how language models process text. The vulnerability was named EchoLeak, assigned CVE-2025-32711, and it serves as a useful study to understand the landscape of LLM security.
For contrast, consider a different case. For roughly $20 in API queries, a team extracted part of a production OpenAI model through its public interface. That sounds like the threat most teams should really fear.
In this article, we try to build a map of the full attack surface that threatens an LLM’s security. With it, a given LLM feature can be located, its exposure points identified, and new threats reasoned about as they appear.
So let’s start with the single most important property.
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
Trust Boundaries
Almost every LLM vulnerability traces to one property. A language model receives instructions and data as a single sequence of tokens, and that sequence contains no marker separating commands from information.
Traditional software keeps the two apart.
For example, a parameterized database query holds the command in one position and the user’s input in another. This means that the text typed into a name field stays text even if it spells out a SQL statement, because the structure of the query enforces the separation.
However, an LLM context has one position for everything. The system prompt defining the assistant’s role, the user’s message, a document retrieved from a database, and the output returned by a called tool all arrive concatenated in the same sequence. Any portion of that sequence can affect the generated output as if it were an instruction, because the model computes each next token from the entire preceding sequence with no separate treatment for any part of it.
This is where prompt injection comes in, which means the supply of instruction-like text so that the model output conforms to those instructions instead of the operator’s intent. It reaches the model by two routes:
The direct route is a hostile instruction typed into the chat box. This is the version most people picture.
The indirect route places the instruction inside the content that the model retrieves during a legitimate task, such as a web page being summarized, a document being read, or an email in a managed inbox.
EchoLeak was an example of the indirect route. The user requested ordinary work, and the attacker’s instructions arrived in an email the user had not opened. In fact, the payload passed through Microsoft’s dedicated cross-prompt-injection classifier, which indicates that input filtering can be porous when used alone.
The property applies to any system that supplies external text to a model. A feature that processes retrieved search results, uploaded files, tickets, or comments carries indirect injection exposure by construction.
Parameterization solved SQL injection by separating code from data at the database boundary. No equivalent exists for natural language, because instructions and information are both expressed as words, and there is currently no reliable method to mark a span of text as inert and have that marking respected during generation. Filtering reduces the problem to some extent but does not eliminate it.
Since the confusion can occur wherever text enters the model or an action leaves it, the next step is to map those points.
Attack Surface
The OWASP Top 10 for Large Language Model Applications is the industry reference for the most critical LLM risks. Its recent edition presents ten separate items. Placed against the path that data takes through an application, those ten items become positions on a single map.
The pipeline runs in stages.
Input arrives from the user. The system frequently retrieves context to ground its answer, a step usually built on a vector database that stores documents as numerical embeddings and returns the ones most relevant to a query. That pattern is retrieval-augmented generation, or RAG. The model then processes the assembled input. It may call tools or other agents that take actions externally. Output returns to the user. Monitoring surrounds the pipeline, and every component within it originates from a supplier, which forms the supply chain beneath the system.
Here are some more details about each stage and the potential risks associated with them:
Input: Direct prompt injection and unbounded consumption, which OWASP defines as uncontrolled resource use capable of running up a victim’s costs, sometimes called denial of wallet.
Retrieval: Indirect injection, vector, and embedding weaknesses. PoisonedRAG, a 2024 study, corrupted a RAG system’s answers by inserting as few as five malicious passages into a knowledge base of millions, reaching a 90 percent success rate on targeted questions.
Model: Training-data leakage, data and model poisoning, and system prompt leakage.
Tools: Excessive agency, the condition of an agent holding more permission than its task requires.
Output: Improper output handling, where the response passes downstream without sanitization, and misinformation, where a confident answer is incorrect.
Supply chain: Compromise of any component feeding the stages above.
The map provides a reference point for figuring out the specifics of an attack. A new attack can be located on it. The relevant questions are where it injects untrusted text and at which stage, and whether it abuses a permission the model should not hold. The answers indicate both the severity and the applicable defense.
Supply chain cannot be cleanly placed on the path, because a compromised model or poisoned vector store affects every later stage at once. For that reason, it appears as its own segment that spans everything.
The map also exposes a mismatch in attention. The threats that generate the most concern and the threats that reach production are different.
Model Attacks
Attacks aimed at the model’s interior, including weight theft, training-data extraction, and training-time poisoning, are quite real, but for most developers, they rank low for initial effort. They tend to be expensive, narrow, or already mitigated by the model provider.
Here are a few examples:
Model theft: As mentioned earlier in the OpenAI scenario about a team that recovered the final embedding-projection layer of production OpenAI models for under twenty dollars and confirmed previously secret hidden dimensions. The result is significant and also bounded. It recovers one layer among many, and the researchers stated that reconstructing a full frontier model through an API remains impractical, since the cost exceeds training an equivalent model. OpenAI received advance notice and modified its API.
Training-data extraction: In late 2023, a group from Google DeepMind and several universities found that prompting ChatGPT to repeat a single word continuously could cause it to emit verbatim fragments of training data, including real contact details, with megabytes recoverable for a few hundred dollars. The privacy implications were serious, and OpenAI filtered the triggering behavior after disclosure.
Poisoning: In 2025, a team from Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that approximately 250 malicious documents were sufficient to install a backdoor in models from 600 million to 13 billion parameters, with the count remaining roughly constant across model sizes. That finding overturned the assumption that larger models require proportionally more poisoned data. The researchers also stated the limitation directly. The backdoor produced only gibberish output on a trigger phrase, which is a low-stakes behavior they described as unlikely to pose significant risk in frontier models.
The graph below shows the placement of various threats on a fear vs frequency scale:
The ranking matters because attention is finite. A team focused on model theft while deploying an agent with broad permissions has addressed a rare attack and left a common one open.
The bounded status describes the present. These attacks rise in priority for teams that host open model weights, fine-tune on sensitive data, or operate their own training pipelines, where the model’s interior becomes the team’s responsibility rather than a provider’s.
The interior attacks sit at the edge of the map. The center, where the model triggers external actions, carries the larger risk.
Excessive Agency
The point at which LLM attacks cause material damage has a specific structure and is identifiable in a system. It is also called the lethal trifecta. It consists of three capabilities held together by a single agent:
Access to private data, such as an inbox, a customer database, or a source repository.
Exposure to untrusted content, meaning anything read from outside, including web pages, emails, and shared documents.
A channel to send data out or act externally, such as an outbound request, a sent message, or a tool call.
An agent holding all three can be directed by injected instructions to transfer private data to an attacker. Model alignment does not remove this exposure, because producing output that conforms to instruction-like input is how the model normally operates.
A few documented cases follow the pattern:
GitHub’s MCP server, MCP being the Model Context Protocol that connects models to external tools and data, has been compromised using malicious issues filed on a public repository to expose data from a victim’s private repositories.
GitLab’s Duo assistant has been supplied a public project containing hidden instructions and made to leak private repository contents.
A Chevrolet dealership chatbot was once manipulated into agreeing to sell an SUV for one dollar.
A crypto trading agent was socially engineered into transferring 55 ETH.
Removing any one of the three capabilities reduces the exposure. The least costly reduction is usually cutting the outbound channel or narrowing what the agent can access, which tends to be cheaper than adding a stronger filter.
Connecting tools through MCP is the most common way an agent acquires the third capability, and the protocol is recent enough that even established servers have shipped injectable configurations. Anthropic’s own official Git MCP server received three injection-related CVEs in 2025. Runtime input is one source of risk. The system’s components are another, and they can be compromised before any request arrives.
Supply Chain
Every model, adapter, vector store, and tool in a stack originates from a supplier, and any of them can arrive compromised. This is the supply chain surface, and it bypasses runtime defenses because the threat is present before input validation runs.
The common mechanism is straightforward.
Many models are distributed as serialized files, and some serialization formats execute code when the file loads. In early 2025, ReversingLabs documented a technique named nullifAI, in which malicious models uploaded to Hugging Face concealed a reverse shell, code that opens a connection back to an attacker, inside a Python pickle file. The file was compressed in a manner that evaded Picklescan, the platform’s scanner, so the model appeared clean and executed hostile code on load.
The scale demands attention. Protect AI, which scans models hosted on Hugging Face, has examined more than four million and flagged roughly 352,000 as carrying unsafe or suspicious issues, across more than fifty thousand models.
Provenance is one of the few fully controllable factors. The selection of which models, tools, and data sources to trust sits with the team, unlike most of the runtime surface.
Two mitigations are advancing:
Safer serialization formats that avoid executing code on load.
Model signing that verifies origin, comparable to signed releases in package ecosystems.
Defense in Depth
No single defense holds all the time, and the supporting evidence is strong enough that the realistic objective changes from preventing every attack to surviving the ones that succeed. The working posture is defense in depth, a set of independent layers arranged so that the failure of one is contained by another.
One layer is insufficient.
In November 2025, a team from OpenAI, Anthropic, and Google DeepMind published a study that took twelve previously proposed defenses against prompt injection and jailbreaking and defeated them, using attacks permitted to adapt and iterate. Strong production filters still allow a measurable fraction of attacks through, and a single success is enough. A lone guardrail provides confidence that the measurements do not support.
The more durable approach constrains the system around the model rather than relying on the model to resist manipulation.
Google DeepMind’s CaMeL is one example. It treats the model as untrusted, uses a separate privileged component to plan actions, and quarantines externally retrieved data so that data cannot trigger sensitive operations on its own.
Meta’s Agents Rule of Two is a simpler operational version, recommending that an agent satisfy at most two of three risky properties: processing untrusted input, holding sensitive access, and acting externally, without a human in the loop. Meta presents the rule as a supplement to least privilege rather than a complete solution.
The standard layers each cover a stage of the map:
Input is validated and constrained.
Retrieval sources are kept clean.
Each tool is scoped to the minimum permission its task requires.
Model output is treated as untrusted and sanitized before downstream use.
Monitoring watches for anomalies.
A human reviews the highest-consequence actions.
Conclusion
The threat model organizes around one fact.
A language model processes instructions and data as the same sequence of tokens, and the full map follows from that property.
The OWASP Top 10 becomes a set of positions on a path rather than a list. The widely publicized attacks, model theft, and training-data extraction are bounded and largely mitigated, while the larger risk concentrates wherever an agent holds private data, untrusted content, and an external channel at once. The supply chain underlies every stage and is the surface most directly controlled by the team operating the system. Since no single layer holds, defense in depth is the realistic posture to implement for a team.
The cost is high. Every layer adds latency, expense, and friction, and the strongest mitigation, human review of consequential actions, also limits how autonomously a system can operate. The trade-off is unavoidable.
The landscape reduces to a few durable points:
The root cause is the absence of a boundary between instructions and data.
Named threats are positions on the pipeline rather than isolated facts.
The interior attacks are mostly bounded, and the trifecta marks where real damage occurs.
Provenance is the most directly controllable surface.
Defense in depth, rather than any single filter, contains the residual risk.
References:
EchoLeak: zero-click prompt injection in a production LLM (case study)
PoisonedRAG: knowledge-base poisoning of retrieval-augmented generation
New prompt injection papers: Agents Rule of Two and The Attacker Moves Second
Agents Rule of Two: a practical approach to AI agent security
Anthropic Git MCP server injection advisory (CVE-2025-68143)
Did you use AI to write this, or help?
awesome read. great to see security fundamentals being covered!