Choosing a vendor for AI in service? How to judge a platform on what matters

The observability gap: can you see what your agent is doing? 

Monitoring and observability have the most direct impact on whether a deployment keeps delivering results over time. Good observability doesn't just show you what happened in any given interaction. It shows you why it happened and what would happen if you changed the inputs. 

Most platforms offer dashboards tracking containment, escalation rates and conversation volume. These are useful, but not enough. They tell you what happened, not why, and won't flag when a routine model update has quietly degraded performance on a particular intent – something that can go unnoticed for weeks if you're only watching top-line numbers. 

All this means you should treat observability as a contractual requirement, not a nice-to-have to be sorted out later. Ask vendors exactly what data you can extract, in what format and at what granularity. Press on what SLA applies to detecting an incident.  

Ownership after go-live: who gets to make changes?

An AI agent doesn't deliver its value on day one, but over the following months as someone maintains, extends and improves it. The question worth asking a vendor directly is: who does that, and how? 

On some platforms, every change, however small, has to go through a developer, which means every improvement waits in a technical queue. On others, a business analyst or a contact centre manager can make updates directly, without waiting on anyone. That difference sounds minor at the shortlisting stage. It compounds, in both cost and speed, for as long as you're running the platform. 

This is also about independence. Ownership after go-live determines whether you can build on the agent incrementally, on your own terms, or whether you stay dependent on the vendor for every meaningful change. In our experience, the freedom to improve the agent post-launch shapes total cost of ownership more than almost anything else. It's also the most underweighted criterion in vendor RFPs, because it's invisible in a demo and only becomes obvious about six months into a live deployment.

Who sets the limits on what it's allowed to do?

Ownership over changes is one form of control. The other is knowing when the agent should stop acting on its own. Is the vendor explicit about how an agent's autonomy threshold gets set? How far can the agent go before it has to bring in a human? This is the difference between an agent that quietly oversteps its remit and one that knows precisely when to step back. This is particularly important in regulated environments, where the consequences of getting the threshold wrong are both legal and reputational. 

Just as important is who owns the customer data flowing through the agent, where it is stored and how the vendor is permitted to use it. Press for explicit answers on data residency, whether any customer data is used to train the vendor's shared models, and how you can trace and retrieve your data if you exit. 

In our work with a UK regulator on an AI-enabled complaints process, these weren't details we bolted on after the fact. They shaped which platforms were in the running from the outset. The lesson holds regardless of sector: know where your regulatory lines are before you start shopping, not after you've picked a vendor and discovered the platform can't sit inside them. 

The best platforms are the ones you can trust for the long haul

What separates a genuinely good platform from a merely impressive one is everything that happens after the demo. Can you see clearly into what your agent is doing and why? Can you shape and improve the agent without staying permanently dependent on the people who sold it to you? Does the agent know exactly when to hand control back to a person? 

The best AI agents are the ones you can operate, improve and trust month after month. Choosing the right vendor – for the right reasons – is what makes that possible.