Prompt injection remains the number one security risk for LLM applications according to the OWASP Top 10, yet fool-proof prevention methods remain elusive. Litmiq has released a practical guide featuring twelve single-turn test cases designed to help developers identify vulnerabilities in their system prompts before users exploit them. The approach relies on embedding a unique, meaningless canary string within the system prompt and running specific attacks to see if that string leaks into the model's output.

The Canary String Method

The core technique involves inserting a specific string, such as LMQ-CANARY-7731, into the system prompt with instructions to never reveal, quote, or describe it. This transforms the ambiguous question of whether a model leaked its instructions into a binary search problem. If the canary string appears in any model output, the defense has failed. The guide emphasizes using a made-up string rather than a real secret, as test prompts are often pasted into browser-based tools that send data to model providers.

Twelve Test Cases and Attack Patterns

The suite includes ten attack cases and two benign controls, covering direct injection, indirect injection simulation, and obfuscation techniques. Key tests include PI-01, which uses the classic 'Ignore all previous instructions' override, and PI-05, which hides a command in Base64 encoding. Other cases test role-play persona switches, authority claims from fake developers, and format hijacks where a hostile JSON structure is requested alongside a legitimate question. The two benign controls ensure that the security rules do not become so strict that they refuse normal user queries, such as asking for instructions on connecting a bank account.

Interpreting Results and Limitations

Developers are advised to search outputs for the canary string first, as automated judges can be fallible. The guide notes that partial leaks, such as revealing the length or starting characters of the code, count as failures. However, these tests are limited to single-turn interactions and text-only outputs; they do not cover multi-turn attacks, tool use, or actual indirect injection via retrieval systems. The guide stresses that passing these twelve cases is a smoke test, not a proof of security, and that real-world attackers will use more complex vectors.

Key Takeaways

  • Use a unique, meaningless canary string to make leak detection a simple search task.
  • Include benign controls to ensure security rules do not over-refuse legitimate queries.
  • Recognize that partial information leaks, such as code length or prefixes, are still failures.
  • Understand that these single-turn tests do not cover multi-turn attacks or tool-based actions.

The Bottom Line

Litmiq’s suite is a necessary hygiene check, not a security shield. Developers must treat these tests as a baseline for prompt regression, recognizing that true safety requires architectural mitigations like least-privilege access and human approval for high-risk actions.