Evaluating an AI chatbot requires more than checking whether a few answers sound convincing. This guide provides a reusable testing checklist, scenario-based test cases, scoring rubric, and regression workflow for measuring factuality, task completion, safety, tone, consistency, and escalation before and after a chatbot change.
Overview
An AI chatbot can appear helpful while still failing in ways that matter: it may invent an answer when information is missing, misunderstand a customer’s goal, expose data through an unsafe workflow, or give a different result after a prompt or model update. A reliable chatbot testing process makes these failure modes visible and gives the team a consistent basis for deciding whether a release is ready.
Start by defining what the chatbot is expected to do. Write down its supported intents, connected systems, source-of-truth content, escalation rules, and prohibited behaviours. A customer support chatbot, internal knowledge assistant, website AI assistant, and voice interface will need different test cases. Do not use a single score to represent all of them.
A useful evaluation has four layers:
- Capability: Can the chatbot understand the request and complete the intended task?
- Quality: Is the response accurate, relevant, clear, and appropriately concise?
- Safety: Does it protect sensitive information, resist manipulation, and refuse or redirect unsuitable requests?
- Operations: Does it respond within acceptable limits, call tools correctly, log useful evidence, and escalate when required?
For structured outputs, validate the response mechanically rather than relying only on a reviewer’s judgement. The guide to reliable JSON prompting explains how schemas, validation, retries, and test templates can support this approach.
Checklist by scenario
1. Normal user requests
Build a representative set of questions from real support themes, product documentation, internal workflows, or designed examples. Include different phrasings, spelling errors, short messages, long messages, and follow-up questions.
- Does the chatbot identify the correct intent?
- Does it use the approved source or tool instead of guessing?
- Does it answer every part of a multi-part request?
- Is the response understandable to the intended audience?
- Does it ask a useful clarifying question when the request is ambiguous?
For each case, record the expected outcome rather than insisting on one exact sentence. Several responses may be acceptable if they contain the required facts, follow the correct process, and meet the tone and safety requirements.
2. Knowledge and retrieval questions
For a retrieval-augmented generation, or RAG, chatbot, test questions whose answers are clearly present in the knowledge base, partly present, outdated, or absent. Check whether the chatbot distinguishes between these situations.
- Does it cite or identify the relevant source where your interface requires this?
- Does it avoid adding details that are not supported by retrieved content?
- Does it state when the available information is insufficient?
- Does it handle conflicting or duplicate documents predictably?
- Does it avoid treating instructions inside retrieved text as higher-priority instructions?
Keep a small set of deliberately unanswerable questions. A chatbot that says it does not have enough information can be performing better than one that produces a confident but unsupported answer.
3. Tasks, tools, and API actions
When the assistant can create tickets, search records, update an account, or call an external API, test the complete workflow rather than only the natural-language response. Confirm that the chatbot collects required fields, validates values, and handles tool errors without claiming that an action succeeded when it did not.
- Use a valid request with all required information.
- Omit one required field and check whether the chatbot asks for it.
- Provide an invalid value, expired identifier, or unavailable option.
- Simulate a timeout, permission error, duplicate request, and partial failure.
- Check whether confirmation is required before consequential actions.
The API and webhook integration guide is useful when turning these cases into a repeatable tool-testing plan.
4. Safety, privacy, and prompt injection
Test direct attempts to override system instructions, requests for secrets, extraction of hidden prompts, and instructions embedded in user-provided documents or web content. Also test whether the chatbot reveals personal or confidential information from conversation history, retrieved documents, or tool results.
Assess the behaviour, not just whether the model refuses. A good response should avoid the unsafe action, provide a brief explanation where appropriate, and offer a safe alternative or escalation path. Review prompt injection testing examples when expanding this test set.
5. Escalation and difficult conversations
Include frustrated users, repeated failed attempts, complaints, vulnerable situations relevant to the product, and requests that require a human or specialist team. Check whether the chatbot recognises its limits, preserves useful context, and transfers the conversation without making the user repeat everything unnecessarily.
Scoring rubric and regression workflow
Use a simple per-case rubric so reviewers can explain why a response passed or failed. A five-point scale works well when the definitions are explicit:
- 5 — Strong: Correct, complete, safe, appropriately worded, and operationally successful.
- 4 — Acceptable: Meets the goal with a minor wording issue or non-critical omission.
- 3 — Borderline: Partly useful but needs correction, extra clarification, or a manual check.
- 2 — Poor: Misses an important requirement, uses an unreliable process, or creates avoidable confusion.
- 1 — Critical failure: Fabricates material facts, performs an unsafe action, exposes restricted information, or gives a harmful escalation result.
Score separate dimensions for factuality, task completion, relevance, tone, refusal behaviour, and escalation. Mark critical failures separately rather than allowing a high average to hide them. For example, a chatbot should not pass a release because it is polite if it also incorrectly confirms account changes.
Store each test case with an ID, scenario, input, expected behaviour, required facts, forbidden behaviours, model or prompt version, actual output, scores, reviewer notes, and pass/fail decision. Run the same fixed set before and after every material change to the prompt, model, retrieval configuration, tools, safety rules, or knowledge base.
Compare both aggregate results and individual regressions. A new version may improve factuality while becoming more verbose, or improve task completion while weakening refusal behaviour. Automated checks can flag missing fields, unsupported citations, invalid JSON, tool-call errors, and forbidden phrases. Human review remains important for nuance, tone, ambiguity, and acceptable alternatives. For production evidence, connect test results with logging and tracing practices described in the chatbot observability guide.
What to double-check
- Test diversity: Include paraphrases, typos, incomplete questions, different user roles, and realistic conversation history.
- Context limits: Test long conversations, large retrieved passages, and requests near your configured token budget. The token limits guide covers practical checks for context and cost control.
- Version control: Record the exact prompt, model configuration, tools, retrieval settings, and knowledge-base revision used for each run.
- Tool truthfulness: Confirm that the assistant distinguishes between a requested action, a failed action, and a completed action.
- Human agreement: Give the same sample to more than one reviewer or calibrate reviewers against a shared set of examples.
- Voice behaviour: For voice AI, test recognition of accents, interruptions, silence, barge-in, confirmation prompts, and speech synthesis clarity. The voice assistant design guide provides a related workflow.
Common mistakes
Testing only happy paths. A small collection of ideal questions cannot reveal ambiguity, tool failures, unsupported requests, or unsafe behaviour.
Measuring fluency instead of usefulness. A confident and well-written answer can still be factually wrong. Score against expected outcomes and approved sources.
Changing the test set after every failure. Keep a stable core suite so results remain comparable. Add new cases to a separate expansion set, then promote important cases into the core suite.
Using one overall pass rate. Separate critical safety and privacy failures from ordinary quality issues. Set release gates for both.
Ignoring conversation state. Test follow-ups, corrections, topic changes, and repeated questions. A chatbot may pass isolated prompts while losing information across turns.
Failing to version prompts. Treat prompts as production artefacts. The prompt versioning workflow can help teams review changes, compare results, and roll back safely.
When to revisit
Run the core chatbot testing checklist before launch, after any material prompt or model change, and whenever you update tools, retrieval logic, safety controls, or source content. Re-run relevant scenarios after investigating negative user feedback, failed escalations, incorrect answers, or unexpected costs.
Before seasonal planning cycles, add temporary demand, policy, catalogue, or support scenarios to the test set if those inputs affect the chatbot. When workflows or tools change, test both the new path and the old path to identify regressions. Review the full suite on a regular schedule, such as quarterly, and remove cases only when the underlying requirement no longer exists.
For the next review, take these practical steps:
- Export the current prompt, model, tool, and knowledge-base versions.
- Select a balanced set of normal, ambiguous, unsafe, tool-based, and escalation cases.
- Score each case against factuality, completion, tone, safety, and escalation criteria.
- Compare results with the previous baseline and investigate every critical failure.
- Record accepted risks, assign fixes, and rerun failed cases before release.
A disciplined evaluation process turns chatbot quality from a subjective impression into an ongoing engineering practice. The test set should evolve with the product, while its core cases remain stable enough to show whether each change genuinely improves the experience.