How to test a WhatsApp AI agent before launch

Check the outcome as well as the words. A good answer explains what is known and stays within the authority you gave the agent. If the agent claims to have changed something, the system that owns that record must confirm it.

Anthropic’s guide to evaluating agents distinguishes a conversation transcript from the actual end state. Apply that distinction to your own workflow: a sent acknowledgement, a received request and a confirmed reservation are separate observations.

Use a separate row for each scenario. Give the person checking the result enough information to reproduce it. Record failures rather than rewriting the expected outcome to fit the answer.

Begin with an ordinary successful request, then change one condition at a time. Use test records and a controlled environment for actions that could create a booking, charge a customer or send a message. Do not perform a real transaction just to make a demonstration look complete.

Illustrative request: “Do you have a room for our family next Friday?”

The agent should clarify the actual dates and guest requirements before treating the request as complete. If an authorised availability lookup exists, compare its result with the answer. If it does not, expect the enquiry to reach reservations.

Receiving an enquiry does not confirm a booking. If the agent says a reservation exists, check the hotel’s reservation record. Also try a sold-out date, a changed departure date and a failed lookup. The hotel WhatsApp guide explains the operational journey that these tests should support.

For an order-status question, compare the reply with the correct order record. For an order change, verify both permission and the final state. A lookup permission should not be treated as permission to cancel an order or issue a refund. Use the ecommerce guide to choose realistic order exceptions.

For clinic appointments, keep the test administrative: dates, location, scheduling and a path to staff. Do not turn a scheduling test into diagnosis or medical advice. The clinic guide keeps that boundary explicit.

First identify whether the problem is in the information, the decision, the tool result or the receiving process. Correct an outdated answer in its source. Fix a permission or routing problem where the action is configured. A longer prompt cannot make an unavailable record current.

Use the knowledge-preparation checklist for source failures and the handoff guide for ownership failures. Re-run the failed case and related cases after the correction.

Repeat relevant tests after changing instructions, business information, models, tools or permissions. Save the previous results so you can identify a regression. Some model responses vary, so a single successful attempt should not be treated as proof of consistent behaviour.

Passing your test sheet is useful evidence for the tasks it covers, not a guarantee that every future conversation will be correct. After launch, review failures and add representative new cases. The automation measurement guide helps separate useful customer outcomes from activity counts.