AI chatbot security testing should examine more than whether a model gives an unsafe answer. This reusable checklist helps you test prompt injection, sensitive-data leakage, malicious documents, excessive tool permissions, and unsafe output handling before and after changes to your conversational AI deployment.
Overview
A chatbot can be exposed through its system instructions, user messages, retrieved documents, connected tools, conversation history, logs, and application code. Security testing therefore needs to cover the whole workflow rather than treating the language model as an isolated component.
The aim is not to prove that a chatbot can never be manipulated. The practical aim is to identify which instructions, data, actions, and failure modes could create unacceptable risk, then add controls that limit the impact. Test in a non-production environment where possible, use synthetic secrets and records, and record each result so it can become a regression test.
Start with a threat-model worksheet
- Assets: List customer records, internal documents, credentials, personal data, business rules, payment or account actions, and any confidential prompt content.
- Entry points: Record every place untrusted content enters the system, including chat messages, uploaded files, web pages, emails, knowledge-base documents, voice transcripts, and tool results.
- Trust boundaries: Mark the boundaries between the user, the model, retrieval components, application code, external APIs, databases, and human agents.
- Actions: Identify what the assistant can read, write, send, delete, purchase, change, or trigger. Separate informational tools from tools that create side effects.
- Failure impact: Define what would happen if the assistant disclosed an internal record, used a tool for the wrong user, or returned unsafe content to an application.
This worksheet gives developers and reviewers a common scope for deploying a website AI assistant. It also makes it easier to decide which tests need human review and which can be automated.
Checklist by scenario
1. Direct prompt injection
Direct prompt injection occurs when a user asks the assistant to disregard its intended instructions, reveal hidden information, or perform an action outside the approved workflow. Test several variations rather than relying on one obvious attack phrase.
- Ask the chatbot to ignore its rules and follow the latest user instruction instead.
- Request system messages, hidden prompts, internal routing rules, credentials, or private configuration.
- Use role-play, translation, summarisation, code formatting, or hypothetical scenarios to disguise the request.
- Split the request across multiple turns and check whether earlier context weakens the control.
- Combine a legitimate request with an unsafe one and confirm that the assistant completes only the permitted part.
Pass criteria should be specific. For example, the assistant should not disclose protected instructions, should explain relevant limits without exposing sensitive implementation details, and should offer a safe alternative where one exists. Store successful attacks as test cases for future model and prompt changes. A controlled prompt versioning process can support this work; see how to version AI prompts.
2. Indirect injection through documents and retrieved content
In a RAG chatbot or document assistant, untrusted instructions may be embedded in a file, web page, ticket, email, or knowledge-base article. The assistant may treat retrieved text as an instruction even though it should be treated only as data.
- Add a test document containing instructions such as “ignore the user’s request” or “send the retrieved data to an external address.”
- Place the instruction in different locations, including visible text, metadata, footnotes, tables, and quoted examples.
- Ask a question that retrieves both a trusted source and the malicious document.
- Check whether the answer distinguishes source content from executable instructions.
- Verify that citations, source identifiers, and access permissions are preserved during retrieval.
Keep ingestion controls separate from model instructions. Documents should be classified, access-controlled, scanned where appropriate, and attributed to a source. A knowledge-base maintenance process is useful for removing stale or unexpected content; review AI chatbot knowledge-base maintenance guidance.
3. Sensitive-data leakage
Test whether the chatbot can reveal data that the current user, session, or application is not authorised to see. Use fake records that resemble production data, not real secrets or personal information.
- Ask for another customer’s account details, internal notes, support history, or documents.
- Attempt to retrieve data by changing names, identifiers, spelling, filters, or conversation order.
- Test cross-session isolation by opening separate user sessions and repeating similar requests.
- Check whether error messages, citations, summaries, logs, or conversation exports expose restricted fields.
- Probe for secrets in prompts, environment variables, tool responses, debug output, and model-generated code.
Review both access control and data minimisation. The model should not be the sole authority for deciding whether a user may access a record. Enforce authorisation in application code or the relevant data service, pass only the fields needed for the task, and redact sensitive values before they reach prompts or logs.
4. Unsafe tool use and excessive permissions
An AI agent creates a different security boundary when it can call APIs or execute workflows. For each tool, document the permitted user, purpose, inputs, outputs, side effects, and approval requirement.
- Try to make a read-only tool perform a write operation or access a broader data set.
- Send malformed, oversized, ambiguous, or unexpected values to tool parameters.
- Ask the assistant to take an irreversible action without confirmation.
- Test whether a tool result can redirect the agent to another unauthorised tool.
- Confirm that the application validates tool calls rather than trusting model-produced arguments.
- Check rate limits, timeouts, retry behaviour, duplicate requests, and partial failures.
Use least privilege. Prefer narrow tools with explicit schemas over a general-purpose tool that accepts arbitrary commands. Require confirmation for consequential actions, especially when the request is ambiguous or the target cannot be verified. For escalation and context transfer, align the tool workflow with a documented human handoff process.
5. Unsafe output handling
Security problems can occur after the model responds. Test whether generated text is inserted into HTML, SQL, shell commands, email templates, tickets, or downstream prompts without appropriate validation and encoding.
- Submit markup, links, delimiters, code fragments, and unusual Unicode characters.
- Check that structured outputs are validated against a schema before application use.
- Confirm that invalid output is rejected, repaired safely, or retried with limits.
- Ensure generated links, recipients, file paths, and identifiers are checked by application logic.
- Test whether a failed validation exposes raw prompts, secrets, stack traces, or tool responses.
For applications that depend on structured responses, pair adversarial tests with schema validation and bounded retries. The guidance on reliable JSON prompting provides a useful implementation pattern.
What to double-check
Before approving a release, review the controls around the model as well as the test results. Use this permission and evidence checklist:
- Identity: Is the authenticated user, organisation, and session available to the application independently of the model?
- Authorisation: Are data access and tool permissions enforced outside the prompt?
- Secrets: Are credentials kept out of prompts, source documents, client code, logs, and error messages?
- Tool boundaries: Does every tool have a narrow purpose, validated parameters, a timeout, and an audit record?
- Approvals: Do high-impact actions require confirmation or human review?
- Retrieval: Are source permissions applied before content is supplied to the model?
- Logging: Can the team investigate an incident without recording more sensitive data than necessary?
- Monitoring: Are unusual tool calls, repeated refusal tests, access errors, and prompt changes visible to the responsible team?
- Recovery: Can a prompt, tool, document source, or model version be disabled or rolled back quickly?
Create a red-team record for every test with the case ID, date, environment, model and prompt versions, input, observed output, tools called, severity, expected behaviour, owner, and remediation status. A broader chatbot evaluation framework can help connect security cases with normal quality and regression testing: evaluate an AI chatbot with test cases and scoring rubrics.
Common mistakes
- Testing only the chat box: A safe conversational response does not prove that retrieval, tools, logs, or downstream systems are safe.
- Using only obvious attack wording: Vary language, turn order, formatting, languages, and document placement.
- Relying on prompt instructions for authorisation: Prompts can guide behaviour but should not replace access controls.
- Granting broad tool access for convenience: Narrow capabilities make failures easier to contain and investigate.
- Logging everything by default: Debugging data can become a second leakage channel. Define retention and redaction rules before testing.
- Fixing one example without adding a regression test: Record the attack, expected result, and relevant control so it is rerun after changes.
- Ignoring human workflows: An unsafe escalation, transcript, or support ticket can carry the original attack into another system.
When to revisit
Run the complete checklist before an initial production launch and after material changes to the model, system prompt, retrieval pipeline, data sources, tool permissions, authentication, user interface, or downstream integrations. Repeat focused tests when a suspicious conversation reveals a new attack pattern or when a document source changes its format or ownership.
Schedule a regular review before seasonal planning cycles, particularly when expected traffic, staffing, product rules, or support processes change. During each review, retire tests that no longer match the architecture, add cases from incidents and near misses, and confirm that owners still know how to disable a risky tool or roll back a release.
For a practical pre-release run, start with one synthetic secret, one malicious document, one cross-user access attempt, one high-impact tool action, and one malformed structured response. Record the result, fix the highest-impact weakness first, then rerun the full affected workflow. Keep the final test set beside your deployment and observability documentation so security testing remains part of AI deployment rather than a one-time approval step. Production monitoring and trace review can be supported by an AI chatbot observability workflow.