LLM Penetration Testing: Model, RAG, Tool and Data Boundaries

Last updated: November 4, 2026.

LLM penetration testing evaluates the application around a model, not only the model’s willingness to produce an unsafe answer. Real systems add retrieval, memory, identity, tools, file handling, web access and business APIs. A weakness becomes operationally important when model output crosses one of those boundaries without the expected authorization or review.

The scope should separate model behavior from deterministic application controls. Prompt injection, data leakage and unsafe tool use may interact, but they require different evidence and owners. A useful engagement records the input, retrieved context, policy decision, tool request, actual side effect and final response.

For the assessment framework, consult OWASP Top 10 pour les applications LLM and map the relevant guidance to the actual environment and authorized scope.

Map the LLM system as a sequence of trust decisions

Document every component that can add instructions, data or capability. Include system prompts, user content, retrieved documents, memory, plugins, MCP servers, internal APIs, browser automation, code execution and human approval points.

Test prompt injection as a control problem

A prompt injection string alone is not a complete finding. Demonstrate which instruction boundary failed and whether the result changed data access, tool use, workflow state or trusted output. Use synthetic data and reversible actions whenever possible.

• Direct and indirect instructions from documents, websites, messages and tool results.
• Instruction hierarchy, task boundaries and context isolation.
• RAG access controls before retrieval, not only redaction after generation.
• Tool argument validation, allowlists, confirmation and least privilege.
• Cross-user, cross-tenant and cross-session memory isolation.

Measure repeatability and operational impact

Generative behavior varies. Record model and application versions, temperature or sampling controls when available, test fixtures, number of attempts and success criteria. A finding should distinguish a reproducible application-control failure from a low-confidence content-policy observation.

Retest the deterministic boundary

A remediation may change prompts, retrieval filters, tool permissions, approval logic or output handling. Retest the original case and adjacent variants, then confirm that legitimate tasks still work. Store the regression cases so future model or orchestration changes can be evaluated consistently.

Questions buyers should ask

• Does the scope include RAG, tools, memory and downstream APIs?
• How are model versions and nondeterministic results recorded?
• Can the provider distinguish unsafe text from an unauthorized side effect?
• Will the final test set be reusable after a model or prompt change?

Conclusion

LLM penetration testing is most valuable when it follows data and authority through the entire system. The deliverable should identify the failed boundary, demonstrate a controlled impact and leave the engineering team with repeatable cases that verify the remediation.

Sources and standards

• OWASP Top 10 pour les applications LLM
• Cadre de gestion des risques liés à l'IA du NIST
• MITRE ATLAS