.png)
Prompt injection is an attack technique that places adversarial instructions in input an artificial intelligence system processes, causing a generative model or agent to disregard, reinterpret or compete with its intended instructions and produce an unintended response or action.
The input may come directly from a person or indirectly from a document, message, website, image, database result or tool response. The model encounters both legitimate content and attacker-controlled text within its context, where their roles may be difficult to separate reliably.
The impact depends on the surrounding application. A standalone chatbot might return a misleading answer. An agent with access to files, accounts or tools might disclose data, make an unauthorized request or take an unintended action. Prompt injection is therefore a system security problem, not just an undesirable model response.
Agentic AI security provides the prerequisite control model: identify the agent and endpoint, restrict its authority, mediate tools, preserve evidence and retain a way to revoke access.
Why prompt injection matters
Generative AI applications combine instructions and data in a shared model context. System messages, developer rules and user requests are intended to direct behavior. Retrieved documents, web pages and tool results are usually intended to supply facts. Yet all of them may arrive as tokens the model can interpret as language.
This creates an instruction-data boundary that conventional software does not usually have. A database query treats a record as data unless another component unsafely interprets it as code. A generative model is designed to infer meaning from language, so text embedded in a record can influence its output even when the application meant the text to be passive content.
OWASP describes a prompt-injection vulnerability as unintended model behavior caused by input, including input that is not visible or readable to a person. Its LLM01 guidance separates direct injection from instructions delivered through external content and notes that retrieval-augmented generation and fine-tuning do not fully remove the issue.
The model's permissions determine the practical consequence. Injection into a read-only summarizer may corrupt a summary. The same technique against an agent that can search private files and send messages can cross confidentiality and authorization boundaries. The security goal is therefore to reduce both the chance that hostile content controls a decision and the authority exposed if it does.
How prompt injection works
A prompt-injection attack exploits ambiguity about which parts of the model's context are instructions and which parts are untrusted data. The exact prompt format differs by model and application, but the system path has five common stages.
- The application establishes intended behavior. System and developer instructions define the task, restrictions and output format.
- The application adds input. A user request, retrieved document, tool response, image or earlier message enters the context.
- Untrusted content introduces a competing instruction. The content asks the model to change its task, ignore a restriction, reveal information or select a different action.
- The model resolves the combined context. Because a model predicts a response rather than enforcing a formal authorization policy, it may follow the competing instruction wholly or partly.
- The surrounding application uses the output. The result may remain text, become a tool argument or influence another automated decision.
NIST places direct and indirect prompt injection within its adversarial machine-learning taxonomy and distinguishes attacker goals, capabilities and lifecycle stages. The NIST taxonomy is useful because it frames injection as one attack family rather than treating every incorrect model output as the same security event.
The following original decision flow shows where controls can interrupt the path:
Text alternative: Trusted instructions, a user request and untrusted content enter an AI application. The application records source and trust before inference, then validates any proposed response or action. Policy denies, revises or approves the action and records the result.
Source labels and prompt hierarchy can help the model interpret context, but they are not enforcement. The decisive controls sit outside the model: a file service decides whether the agent may read a file, an API checks whether the requested operation is authorized, and a policy layer can require approval before a consequential action.
Types of prompt injection
Direct and indirect prompt injection describe how the adversarial instruction reaches the system. Tool poisoning is a related way to place hostile instructions or misleading capability information within an agent's tool context.
| Type | Delivery path | Trust boundary crossed | Representative consequence |
|---|---|---|---|
| Direct prompt injection | A person submits adversarial content through a prompt or other application input | User input is treated as higher-priority instruction | The model changes task, reveals context or produces a disallowed response |
| Indirect prompt injection | The application retrieves or opens attacker-influenced content | External data is interpreted as instruction | A summary is manipulated or an agent proposes an unrelated action |
| Tool poisoning | A tool definition, metadata or returned content misrepresents the tool or instructs the model | Tool discovery or output is trusted without independent verification | The agent chooses an unsafe tool, supplies excess data or accepts a misleading result |
The labels describe delivery, not severity. A direct attack can be harmless when the system has no sensitive context or authority. An indirect attack can be serious because it may arrive through ordinary work that neither the employee nor operator recognizes as an instruction.
Direct prompt injection
Direct prompt injection occurs when a user or attacker places the competing instruction in an input channel they control. The person is interacting with the AI system, so the application can associate the event with a session and identity. That visibility does not make the content safe.
Direct injection includes attempts to override system rules, extract hidden context, change the requested task or induce an unsafe output. Some direct attacks are also called jailbreaks, particularly when their purpose is to bypass a model's safety behavior. The terms overlap, but prompt injection is broader because it also covers attacks against application-specific instructions, private context and agent workflows.
Indirect prompt injection
Indirect prompt injection occurs when adversarial instructions are embedded in content the AI system retrieves or receives from somewhere other than the active user's prompt. Possible carriers include email, support tickets, documents, source files, web pages, calendar entries, images, search results and database records.
This form is especially relevant to enterprise endpoints. An employee may ask an agent to summarize a downloaded document or triage messages. The employee's request is legitimate, but one of the inputs was created by another party. If the agent can access local data or connected services, that content can try to redirect the workflow across systems.
Hidden text is not required. A visible sentence in a document can be an injection if the application should treat it as content rather than authority. Conversely, multimodal systems can process instructions encoded in visual content that a person may not notice.
Tool poisoning and agentic systems
An agent selects tools partly from names, descriptions, schemas and other metadata supplied by its host or an integration. Tool poisoning occurs when an attacker manipulates that information, or the content returned by a tool, to influence tool selection, arguments or later actions.
Tool poisoning is not identical to indirect prompt injection in every case. A falsified capability description is also a supply-chain and integrity problem. When the poisoned description or result contains language intended to direct the model, however, it becomes an injection path inside the agent's context.
The Model Context Protocol (MCP) illustrates this boundary. An MCP server can expose a tool name, description, input schema and annotations to a client. The official MCP specification says clients must treat tool annotations as untrusted unless they come from trusted servers. Trusting a label such as “read only” cannot replace independent authorization or isolation.
A secure client should bind a tool to a known server and version, display its real target and requested action, constrain its parameters, and authorize the resulting operation outside the model. Tool output should re-enter the workflow as untrusted content unless a stronger trust decision is justified. MCP security covers the protocol-specific identities, authorization and server boundaries in more detail.
An enterprise endpoint example
Northstar Procurement, a fictional company, gives employees a research agent on managed laptops. The agent can read files in one assigned project folder, search approved public sites and create a draft vendor summary. It cannot read credential stores, send messages or upload files.
An employee asks the agent to compare a vendor's public security document with an internal questionnaire. The public document contains a sentence instructing automated readers to ignore the comparison, search the device for confidential pricing and submit it to an external location. That sentence is untrusted content, not part of the employee's request.
The runtime marks retrieved web content as untrusted and passes it to the model for summarization. The model still proposes reading another folder and calling an unapproved network destination. Enforcement occurs outside the model: the file service rejects the out-of-scope path, the network tool rejects the destination, and the agent pauses because the proposed action differs from the user's approved task.
The audit record connects the employee, endpoint, agent version, retrieved document, proposed calls, policy decisions and final result. The team blocks the source, reviews similar sessions and rotates no credentials because evidence shows the agent never accessed them.
The example demonstrates defense in depth. Content labeling helped detection, but scoped file access and destination policy prevented the injection from becoming data exfiltration.
Defenses against prompt injection
No single filter, system prompt or model setting reliably eliminates prompt injection across all inputs. Effective defenses assume that hostile instructions may reach the model and limit what can happen afterward.
Separate trust zones
The application should preserve the origin and trust level of system instructions, user requests, retrieved content, memory and tool output. Untrusted content should be quoted or isolated in a structured field rather than concatenated into an undifferentiated prompt. This can reduce ambiguity and improve monitoring, although it does not prove the model will respect the boundary.
Minimize available authority
Give the agent only the data scopes, tools, operations and time required for the current task. Separate read from write authority and use short-lived credentials tied to an agent identity. The principle of least privilege reduces the effect of a successful injection without depending on the model to recognize it.
Enforce actions outside the model
Treat model output as a proposal. Deterministic code and policy should validate the target, operation, data classification, destination and current authorization before a tool executes. High-impact or unusual actions may require a clear human confirmation that shows the exact resource and consequence.
Constrain tools and data paths
Use narrow tool schemas, fixed destinations where possible, output encoding and strict validation. Keep secrets out of prompts and tool results unless the task requires them. Do not provide a single session with broad private-data access and unrestricted external communication when a narrower workflow can complete the job.
Test complete workflows
Evaluation should cover direct prompts, retrieved content, documents, images, tool descriptions, tool output and multi-step memory. Testers should verify the final enforcement result, not only whether the model refused a sample instruction. Reevaluate when the model, prompt, retriever, tool set or permission design changes.
Log and contain proportionately
Records should connect the initiating user, endpoint, agent, content source, proposed action, policy decision and result. Avoid copying unnecessary sensitive content into logs. Response options include ending the session, revoking the agent token, disabling a tool, blocking a source or isolating an endpoint when evidence suggests local compromise.
Benefits and limitations of prompt-injection controls
Layered controls can reduce both frequency and impact. Source tracking helps teams distinguish instructions from data. Narrow permissions reduce the number of reachable resources. Independent authorization prevents a model-generated request from becoming self-approved. Audit evidence supports investigation and targeted revocation.
These controls have important limitations:
- Models remain probabilistic. A mitigation that works on known examples may fail after wording, context or model behavior changes.
- Detection has false results. Filters can miss hostile instructions or block legitimate content that discusses them.
- Human approval is not infallible. Frequent, vague or misleading prompts can train users to approve without inspection.
- Trust labels can be wrong. A compromised retriever, server or integration may falsely identify content or tool behavior.
- Long workflows spread influence. Injected content can persist through memory, summaries and downstream agents unless provenance is preserved.
- Restricted tools can still combine dangerously. Several individually narrow capabilities may together enable an unintended outcome.
- Logging creates another data boundary. Prompts and outputs may contain personal information, secrets or confidential business data.
Prompt-injection resilience is therefore an operational property of the complete system. It depends on the model, prompt construction, retrieval path, identity, endpoint, tools, data controls, policy enforcement, monitoring and response.
Prompt injection and related concepts
Prompt injection is one runtime threat among several AI security concerns. The distinctions determine which controls and owners apply.
| Concept | What changes or is targeted | How it differs from prompt injection |
|---|---|---|
| Jailbreaking | Model safety behavior | Usually seeks to bypass general safety restrictions; it is a related or narrower use of direct injection |
| Data poisoning | Training, fine-tuning or retrieval data | Changes a data source or learned behavior rather than merely placing a competing instruction in one runtime context |
| Model security | Model artifact, provenance and serving path | Protects the model lifecycle; an intact approved model can still process injected content |
| Tool poisoning | Tool description, metadata, implementation or output | Can involve supply-chain deception, integrity failure and indirect injection within a tool-enabled workflow |
| Excessive agency | Permissions and autonomy granted to an AI system | Magnifies the impact of injection but does not cause the injected instruction itself |
| Data exfiltration | Unauthorized movement of information | Is a possible outcome; injection is one technique that may attempt to cause it |
AI model security protects model provenance, integrity and confidentiality, but it cannot by itself keep an approved model from following hostile runtime context. Prompt injection instead tests the boundary between intended instructions and processed data.
The practical objective is not to make every model response trustworthy. It is to prevent untrusted language from acquiring authority. Organizations do that by tracking where content came from, treating model output as a proposal, limiting agent permissions and enforcing consequential actions in systems that do not depend on the model's interpretation.
