Chatbot acceptance testing checks whether a website assistant behaves usefully in the situations your customers actually encounter. A fluent answer is only part of the job. The answer also needs to be grounded in current information, stay within the assistant's role and give the visitor a workable next step.
I would start with a small, written question set before putting a chatbot in front of customers. You can then compare the observed answer with an agreed expectation. This is much more useful than asking a few friendly questions and deciding that the demo looks good.
The approach below is a suggested test method, not a report of measured xbot results. Adapt the examples to your own service and verify the capabilities of your chosen setup.
Write the expected behaviour first
Choose a routine question, such as whether a product can be collected from a particular location. Record the authoritative source and what a good response should contain. If collection depends on the product or location, the assistant may need to ask a clarifying question.
Avoid writing one perfect sentence that every answer must match. You are testing meaning and behaviour. A shorter answer can pass if it is accurate, clear and sufficient for the visitor to proceed.
For a question outside the published information, a useful outcome may be a clear statement that the assistant cannot confirm the answer, followed by the correct contact route. An invented opening time is a failure even if the answer sounds confident.
NIST's AI Risk Management Framework provides a broader structure for managing AI risks through governance, mapping, measurement and management. This article applies the practical idea of evaluating an assistant in context; it is not a certification scheme or an official NIST test.
Build the question set from real work
Ask your support and sales colleagues which questions recur and which mistakes would create the most trouble. Remove customer identifiers before using examples. You need the shape of the question, not someone's private order details.
Include more than easy questions with a sentence-for-sentence answer on your homepage. A useful set covers several situations:
- A routine question with a clear, published answer.
- A question whose answer depends on a missing detail.
- A request that combines two different policies.
- A question about a service you do not provide.
- A request for current account or order information the assistant cannot access.
- A complaint or exception that needs a person.
Keep a separate list of high-consequence situations that your assistant should not resolve on its own. Define these with the person responsible for the service. A website chatbot's ability to explain general information does not make it qualified to decide every individual case.
Test what happens when information is missing
Missing information is a normal operating condition. Ask about an unpublished price, an unavailable delivery date or a product that is not in the approved material. Check whether the assistant identifies the gap instead of completing the answer from a plausible pattern.
Then try an ambiguous question. Can I get it tomorrow could refer to delivery, collection or the start of a service. The useful response may be a short clarification rather than a long explanation of every possible interpretation.
Also check contradictions. If your FAQ says one thing and an old campaign page says another, fix the source material and retest. Prompt wording cannot reliably compensate for a knowledge base that contains conflicting business rules.
Give Finnish and English their own test cases
If your customers use both languages, test both directly. A good English demonstration does not establish that the Finnish version handles ordinary phrasing, inflections or mixed product names well.
Write natural questions rather than mechanically translating every sentence. One customer may ask about toimitusaika and another may write milloin tää ehtii perille. The expected business rule should stay consistent even when the phrasing changes.
Include a conversation that changes language halfway through. Decide what you expect: following the customer's latest language is usually clearer than unexpectedly switching back. Check that important conditions survive the change, especially dates, exclusions and the next action.
You do not need to assume that every language variant must use identical wording. Compare the promise made to the customer. If one answer offers a refund and the other only directs the visitor to support, that difference needs investigation.
Treat handoff as a complete customer journey
Testing a handoff means following it to its destination. Does the contact link work? Does the visitor know whether someone is available now or whether they are leaving a message? Does the receiving team get the context your particular setup is supposed to pass along?
Never infer delivery from the assistant saying that it has contacted someone. Confirm the actual destination with a clearly labelled test message and an agreed recipient. Keep test traffic separate from genuine customer requests.
The same principle applies to failure feedback. W3C's form notification guidance explains the importance of clear success and error messages. In a chatbot journey, the visitor should be able to distinguish a completed action from a suggestion to take one.
If a handoff mechanism is not configured, the assistant must not claim it has created a ticket or sent an email. A plain, working contact route is more useful than a promised action that never happens.
Keep a simple acceptance record
Use a shared table so that fixes can be checked against the original problem. The following rows are illustrative, not actual test results.
| Test situation | Expected behaviour | What to record |
|---|---|---|
| Published opening hours | Answer from the current location page | Source URL, answer and date |
| Unpublished delivery promise | Avoid inventing a date; clarify or route onward | Missing detail and proposed next step |
| Finnish question with English product name | Understand the intent and keep the policy accurate | Language and meaning of the answer |
| Request for private order status | Explain the access limit and use the approved support route | Whether the limit is stated clearly |
| Request for a person | Provide the configured handoff honestly | Destination and actual delivery outcome |
Give each failure an owner and a specific correction. Fix the source, configuration or interface that caused it, then run the case again. Retest related questions as well: changing one policy can affect several answers.
Agree what blocks launch
Define launch blockers before seeing the results. Invented business promises, disclosure of private information, misleading action confirmations and broken contact paths deserve more weight than a slightly awkward sentence.
A single overall pass percentage can hide a serious problem. Keep the individual cases visible and separate factual failures from wording improvements. If a critical case still fails, narrow the assistant's scope or fix the cause before launch.
After launch, keep the question set. Add anonymised examples when customers expose a gap, and rerun relevant cases when policies or source pages change. Chatbot acceptance testing becomes a small maintenance habit rather than a demonstration you cannot reproduce.
If you are planning an assistant for a Finnish business, contact xbot with the routine questions you want it to handle and the situations that should reach a person. That gives the setup and its acceptance test a concrete starting point.