What an AI penetration test should cover in 2026

An AI penetration test should answer a practical question: what can an authorized attacker make this system reveal, change or do? That is broader than prompt injection. It includes familiar application weaknesses, model-specific failure modes and every trust boundary between an AI response and business data or action.

Start with the system, not a vulnerability list

Before choosing test cases, map the people, applications, models, retrieval sources, tools, vendors and sensitive outcomes in scope. A customer chatbot, internal copilot and support agent may use the same model while presenting very different risks.

The initial map should identify:

• who can use the system and how identity is established;
• which data it can read, retain or infer;
• which models and external providers receive requests;
• which tools, APIs or workflows it can invoke;
• which browser, mobile, desktop and backend components complete the product; and
• which outcomes would matter to customers, operators or the business.

1. External attack surface

Begin with what an unauthenticated attacker can discover. Public domains, exposed APIs, abandoned experiments, model endpoints, documentation and vendor integrations can reveal paths the product team does not consider part of the AI system.

Coverage should include asset discovery, service enumeration, public AI features, unauthenticated workflows and hidden or deprecated endpoints. The useful output is a prioritized map of reachable paths—not a long inventory without context.

2. Web, API and business logic

AI features inherit the security of the application around them. Broken object authorization can expose conversation history. Weak sessions can turn a model response into another user’s data. A server-side request weakness in a retrieval connector can bypass every prompt-layer defense.

Test authentication, authorization, session handling, file processing, APIs, rate controls, business logic and third-party integrations where they affect the AI workflow. These controls are often the first link in an AI attack chain.

3. Model and instruction boundaries

Test how the system separates developer instructions, trusted context and untrusted user or retrieved content. Direct and indirect prompt injection challenge that hierarchy, but the finding should describe the boundary that was crossed.

Useful cases include attempts to reveal protected context, bypass a product rule, manipulate a decision, cause unsafe tool use or influence content later consumed by another user. A jailbreak without meaningful impact may deserve attention; it is not automatically a high-severity security finding.

4. Data, retrieval and memory

Retrieval and agent memory introduce data stores that may be poisoned, over-broad or weakly isolated. Test whether a user can reach another tenant’s context, influence shared results, preserve sensitive data beyond its intended lifetime or turn an untrusted document into control instructions.

Include source authorization, document parsing, metadata filters, vector-store isolation, deletion behavior, memory boundaries and tool-returned content. Test both confidentiality and integrity.

5. Agents, tools and action authorization

For an agent, the most important control is usually outside the model: the service that decides whether a tool call is allowed. Test every transition from model output to action. Authorization should use server-side identity and state, never the model’s own explanation.

Probe over-broad grants, parameter manipulation, confused-deputy behavior, cross-agent trust, unsafe retries, unbounded loops and chains where permitted actions combine into an unauthorized result. Stop at the evidence threshold agreed in the test plan.

6. Mobile and desktop clients

Native clients add local storage, operating-system permissions, deep links, inter-process communication and update channels. They may also route model traffic differently from the browser product. Test data at rest, transport configuration, local API exposure, custom URL handlers, plugin boundaries and update trust.

For mobile apps, include backend APIs and device-specific trust assumptions. For desktop apps, include local services, filesystem access, IPC and the relationship between the signed client and remote configuration.

7. Logging, monitoring and failure handling

Security records should help reconstruct an event without creating a second sensitive dataset. Review what is logged, who can access it, whether raw prompts or outputs are retained and how policy decisions connect to identities and versions.

Test unavailable safety services, malformed output, partial tool execution, timeouts, provider changes and missing telemetry. The system should fail according to an explicit business decision, not an accidental default.

8. Reporting and retesting

Each finding should state the affected boundary, prerequisites, reproducible evidence, business impact, practical fix direction and limitations. Where several weaknesses form one exploit path, report that path rather than a collection of disconnected observations.

A retest should reproduce the original path, verify the new control and check adjacent bypasses. The closing statement should say exactly what was verified and what remained outside scope.

Use frameworks as coverage aids

The OWASP GenAI Red Teaming Guide covers model- and system-level risk. The NIST Generative AI Profile supports broader risk and evaluation planning. PTES, OWASP web guidance and mobile standards still apply to the surrounding product.

No framework replaces a system-specific threat model. Use them to find coverage gaps, then prioritize cases by the system’s actual data, permissions and consequences.

A compact scope brief

Start with five facts: what the product does, who uses it, what data it can access, what actions it can trigger and which decision the assessment must support. That is enough to begin shaping authorization, access, evidence and timing.