ORIGINAL TECHNICAL PRACTICE
Treat retrieved instructions as untrusted data
An assistant retrieves a page saying to ignore earlier rules and reveal confidential information. How should the application handle it?
Written for independent practice. The model answer presents one approach; clarify constraints and explain alternatives.
A REASONED APPROACH
Model answer.
Retrieved text is evidence, not an authority to change application instructions or access rules. Enforce permissions and tool policies outside the model; prompting alone is not a complete security boundary.
Work through it
- Separate system policy, the user’s task and retrieved content in the application’s representation. Preserve source labels so the model can distinguish quoted instructions from authorized requests.
- Limit tools to the access needed for the current user and validate tool arguments against explicit allowlists and permissions. Require appropriate approval for consequential actions.
- Test direct and indirect injection attempts, data exfiltration requests and misleading citations. Log safe decision metadata without placing sensitive source bodies in broad-access logs.
Tradeoffs to discuss
- Stricter tool restrictions can reduce convenience. Treat a refusal or safe fallback as an expected result when a request cannot be fulfilled under the user’s permissions.
Common pitfalls
- A document’s claim that the user approved an action is not user approval. Sanitizing a few suspicious phrases does not eliminate the broader injection problem.