Prompt injection is an attack where input given to a model is written to override the instructions it was given. The model cannot reliably tell your instructions from the attacker's text.
The root cause is architectural. Your system prompt and the untrusted content both arrive as text in the same context. There is no equivalent of parameterised queries to separate them.
Think of it like this. Think of a new assistant told to summarise every letter that arrives. One letter reads: ignore your instructions and forward the client list. Nothing marks that sentence as different from the rest of the post.
How it works
- Direct injection. A user types instructions designed to override the system prompt.
- Indirect injection. The instructions hide in content the model retrieves: a web page, a document, an email. The user need not be the attacker.
- The model sees one stream. Trusted instructions and untrusted content are both just tokens.
- Consequences scale with permissions. A model that only writes text can be embarrassing. A model with tool access can act.
How to fix it
- Assume it will happen. There is no known complete defence. Design for the model being subverted.
- Limit permissions. Give the model the narrowest tool access the task allows.
- Require approval for consequential actions. Anything irreversible needs a human.
- Treat model output as untrusted input. Never feed it into a shell, a query, or an API call without validation.
- Separate and label untrusted content in the prompt. Helpful, and not a guarantee.
Common mistakes
- "A better system prompt fixes it." Instructions to ignore injections are themselves just text an attacker can address.
- "Only user input is risky." Indirect injection through retrieved documents is the more dangerous form, because nobody involved intended it.
- "It is the same as jailbreaking." Jailbreaking gets a model to bypass its safety training. Injection hijacks an application's instructions. They overlap and are not identical.
