System prompt extraction, also called prompt leaking, is getting a model to reveal the hidden system prompt that configures it. That prompt often contains guardrail logic, business rules, embedded data, and sometimes secrets like keys or internal URLs. Once leaked, an attacker can design jailbreaks against the exact rules or reuse exposed IP. The defence is to assume the system prompt is semi-public: keep secrets out of it and enforce controls in code, not just in text.
How does system prompt extraction work?
The system prompt and the user message reach the model as one token stream, so the model can be steered to repeat its own instructions. Attackers ask it to print everything above, to summarise its configuration, to translate or encode its instructions, or to continue a partial quote of them. Encodings and roleplay make extraction more reliable when a plain request is refused, because they route around the rule that says do not reveal the prompt.
Why system prompt extraction matters
A leaked system prompt hands the attacker a map. They see the exact guardrails and can craft jailbreaks that target them, and they see any business logic, prompt engineering, or data you embedded. If the prompt holds secrets such as API keys, internal endpoints, or customer data, extraction turns straight into a breach. Prompts are also treated as intellectual property, so leaking one can expose competitive work.
How to defend against system prompt extraction
Treat the system prompt as semi-public. Never put secrets, keys, or sensitive data in it; keep them server-side and enforce authorisation in code. Assume any rule written in the prompt can be read, and back critical controls with real checks outside the model. Add output filtering that detects the prompt being echoed, and red-team extraction through encoding and roleplay.
References
- [1]OWASP LLM01:2025, Prompt Injection(OWASP)
- [2]OWASP Top 10 for LLM Applications(OWASP)
- [3]MITRE ATLAS, Adversarial ML tactics(MITRE)
Assume your system prompt will leak, and make sure nothing sensitive lives in it. Talk to a security expert about testing what yours exposes.