How to Evaluate an AI Receptionist: A Test Plan, Not a Vendor Ranking

Test one: does it complete the job, or just take a message?

This is the first question because it separates the market in half. Ask the AI to book an appointment, and watch what actually happens at the end of the conversation.

Intake-only automation greets the caller, captures a name and number, answers questions from your website, transfers, and takes a message. That is genuinely useful and it beats voicemail. But it means somebody still has to call the person back and find a time, and that step is where leads cool off. In several products, appointment booking is a separate, human-staffed service with its own monthly fee, which tells you where the automation stops.

A product that completes the job ends the call with a confirmed slot on the correct calendar and a confirmation text on its way. Insist on watching the calendar update while you are on the phone. Do not accept a description of it.

Test two: which channels does it actually answer?

Be exact with this word. Answering a channel means a customer can start a conversation there and get a real response. Notifying you on a channel is a different thing, and vendors blur the two.

Many AI receptionists are phone-only products. They may deliver the resulting message to you by SMS or email, which is not the same as a customer being able to text you and get an answer. Conversely, shared-inbox platforms answer text-shaped channels well but do not pick up a ringing phone at all.

Send a text to the number. Open the chat widget. Email the address on the website. Whatever does not answer is a channel your customers are still losing conversations on, no matter what the feature grid says.

• Call it, and ask a question that is not on your website
• Text the same number and see whether anything replies
• Open the web chat and ask the same question again
• Email the published address and time the response
• Then check whether all four landed on one thread for one contact

Test three: what happens when it does not know?

This is the single most predictive test in the list. Ask something reasonable that is not in the knowledge base, and listen closely to the answer.

There are three possible behaviours. It says it does not know, captures the question, and routes it to a person: that is correct, and it is the behaviour you are paying for. It goes quiet, loops, or gives a non-answer: that is a bad experience but a survivable one. Or it improvises something plausible and wrong: that is disqualifying, and no other feature compensates for it.

Then escalate the pressure. Ask for a discount. Ask it to promise a delivery date. Ask it to confirm something legal or clinical. An AI receptionist that will not say no on any of those is not a front desk, it is a liability with a nice voice.

Test four: can a human take over, mid-conversation, on every channel?

Handoff is the feature that earns customer trust, and it is the one most likely to be a badge in a UI rather than a working mechanism.

Take over a live conversation while the AI is mid-reply and verify two things. First, the automation actually stops: if the AI keeps replying after a person claims the thread, the customer gets two answers and you get a complaint. Second, the human sees the full history, including what the AI already said and any actions it took, without asking the customer to repeat themselves.

Check it separately on voice, SMS, chat, and email, because the mechanisms differ and a product can get one right and the others wrong. On voice, the good pattern is a live join so the caller is never asked to hang up and try again.

Test five: can you audit what it did?

You are letting software talk to your customers unsupervised. You need to be able to see exactly what it said and what it did, without asking the vendor.

The minimum is a searchable transcript attached to the contact record, a recording you can listen to, and a log of every tool the AI invoked and what came back. If it sent a booking link, you should be able to see which link and why. If it told a customer something surprising, you should be able to read the sentence.

This matters most in the week after go-live, when you should be reading everything, and again months later when a customer says the AI told them something and you need to know whether it did.

• Searchable transcripts attached to the contact
• Call recordings you can play back
• An audit trail of every tool call and its result
• The ability to see which knowledge source an answer came from
• Export, so the history is yours rather than the vendor's

Test six: the invoice at your volume, and the time to go live

Two commercial tests, and both are commonly skipped until it is too late to matter.

Model the bill at your busiest month, not your average one, and include the channels you are currently ignoring, because the whole point is that they stop being ignored. On metered plans, ask specifically whether robocalls and spam chats consume your allowance, because they will. On flat plans, ask what is passed through as usage.

Then look at how you buy it. If the only way to get the product is to speak to a salesperson and wait, that friction is a preview of the relationship. A small business that just missed three calls this morning does not want to book a call to fix it. Being able to sign up, connect a number, and go live the same afternoon is not a minor convenience; it is the difference between solving the problem this week and thinking about solving it.

Test seven: run it against the calls you actually get

The last test is the one that cannot be borrowed from an article, including this one. Pull your ten most recent real calls, or write down the ten questions you answer most often, and run those through the product verbatim.

Vendors optimise for the generic small business, which is nobody. Your callers ask about a specific service, a specific plan, a specific street, and a specific complication that comes up constantly in your trade. A product that handles the generic script beautifully and fumbles your three most common questions will be abandoned by your team within a fortnight, and the subscription will run for a year.

This also surfaces the knowledge problem early. Most disappointing results are not model failures; they are knowledge failures, where the AI was never given the answer and improvised or deflected. Ask how knowledge is loaded, how it is scoped to a brand, and how you correct an answer that came out wrong. If correcting a bad answer requires a support ticket, you will stop correcting them.

A scorecard you can actually use

Score each product out of seven. Anything that fails the improvisation test is out regardless of its total, because that is the failure that costs you a customer rather than a feature.

We would rather you ran this against us and walked away than bought on a demo and found out in production. We run Contact Center HQ across our own properties, so every item on this list is one we had to pass ourselves.

• Books the appointment, rather than taking a message about it
• Answers voice, SMS, email, and web chat, not just one of them
• Says it does not know, instead of improvising
• Hands off cleanly on every channel, and the AI actually goes quiet
• Gives you transcripts, recordings, and a tool-call audit trail
• Produces a bill you can predict in your busiest month
• Lets you sign up and go live without a sales call