AI automation grew much faster than the criteria for evaluating it. Anyone can now wrap a service around a public API and sell it as an intelligent agent, and from the outside it’s genuinely hard to tell a serious implementation from a chatbot with better copywriting.
These are the questions we’d ask if we were on the other side of the table. Use them on any vendor you’re evaluating, including us.
What happens when the agent doesn’t know the answer?
This is the most revealing one and the one that shows up least in demos. An agent that improvises when it doesn’t know is worse than no agent at all, because it hands your customers false information in the confident tone of something that does know, and the customer has no way to tell the difference.
What you want to hear is an explanation of how the system detects it’s out of scope and how it escalates to a person, conversation context included. Then you should ask to see it live. The red flag here isn’t a bad answer — it’s being told rather than shown: ask to see the failure, not the happy path, because anyone can demo a conversation that goes well.
What systems does it connect to, exactly, and how?
The difference between an agent and a chatbot is that the agent looks up real data. If it doesn’t connect to your calendar, your inventory, or your CRM, all it can do is recite text somebody wrote earlier.
“It connects to Google Calendar via API” is an answer; “it integrates with everything” isn’t. Vagueness here usually means the integration doesn’t exist yet and will be improvised after you sign.
Who owns the data and the conversations?
Your conversation history with customers is valuable business information, and it also contains third-party personal data that you’re legally responsible for protecting. In Colombia that’s Ley 1581 de 2012; wherever you are, something equivalent applies.
You should get a clear answer on four things: that the data is yours, where it’s stored, for how long, and how you export it if you leave. If that answer takes a while or requires checking with someone, you’ve already learned something — a serious vendor has it ready because they’ve been asked before.
What happens if I stop working with you tomorrow?
This is the question that makes vendors most uncomfortable and the one most worth asking, because it determines whether you bought a solution or rented a dependency. What you want to hear is what stays with you — documented flows, exportable data, integrations that keep working — and what switches off.
If everything switches off, the vendor holds your operation hostage, and that quietly changes the negotiation on every renewal for years afterwards.
How do I measure whether it’s working?
Without metrics agreed up front, the three-month conversation about whether it was worth it becomes a debate about impressions, won by whoever talks louder.
You want concrete metrics — response time, share of conversations resolved without a human, enquiry-to-booking rate, no-shows — and a commitment to measure the current state before implementing anything. The red flag is a vendor who doesn’t propose measuring the “before,” because without a baseline any later number is noise presented as a result.
What does it cost to run, not just to build?
The implementation price is what gets negotiated, but the running cost is what you pay forever, and it’s where the surprises usually hide.
Ask specifically about the monthly platform cost, the cost of the AI models by conversation volume, who pays for adjustments when your operation changes, and what happens if your volume doubles. A single price with no breakdown almost always means there are variable costs that will surface later.
Can you show me cases like mine?
Implementing for a clinic looks nothing like implementing for a real estate agency, and experience in your sector cuts setup time and reduces how much has to be figured out along the way.
Here’s something that applies directly to us, so we’ll say it plainly: if a vendor is new and has no case studies, that doesn’t disqualify them. Inventing case studies does. Katalis started recently, and when someone asks us about cases the honest answer is that we’re still building them, so what we can do instead is show the product working. A new vendor who says that up front is a legitimate option; one who talks in the plural about clients that don’t exist is a problem from day one, and it usually shows in the numbers.
Three more signals
Be wary of percentage promises made before the diagnostic, because nobody can promise you a 40% lift in sales without having seen how your operation runs: a number offered on the first call is marketing rather than an estimate. Be wary too of pressure to decide fast — slots that run out today, discounts expiring tomorrow — because a system that will run for years doesn’t get decided in twenty-four hours, and whoever’s rushing you knows that given time you’ll compare. And be wary of the vendor who never tells you no: if you ask about something that doesn’t make sense to automate in your case and the answer is yes to everything, they aren’t evaluating your business.
Before the first meeting
Turning up with five numbers makes the conversation far more useful: how many customer messages you get a month, your average time to first reply, what share arrives outside working hours, your average ticket and close rate, and what systems you use today. If you don’t have them, measuring for two weeks is the best preparation there is, and it’s useful regardless of who you end up hiring — including if you end up hiring nobody.
Of the seven questions, the fourth is the one that defines our business. Katalis comes from catalyst: a catalyst sets off a change that then continues on its own, and if a client can’t sustain the system without us once a project ends, we didn’t do the job properly.
Ask us all seven. We run free 45-minute diagnostics where we review your operation and tell you what can be automated now, what later, and what shouldn’t be automated at all. Get in touch.