RESEARCH · CXS · SUPPORT CHATBOTS
We scored 60 support chatbots with CXS™. 27 scored +50 or higher. 23 had a negative score.
What 600 conversations reveal about the state of AI support chatbots, and how to measure yours in 2027.
In October, isalo's AI agents held 600 conversations with the AI support chatbots of 60 software companies, ten questions each, and rated every answer. We have now used the ratings to compute each chatbot's CXS, the Customer Experience Success Index: the share of interactions that end in full success, minus the share that end in failure, on a scale from −100 to +100.
The result is not a bell curve. Most chatbots either did well or did badly. Few landed in the middle.
60 chatbots · 600 conversations · tested October 8 to 23, 2026 · CXS version 1.0
CXS across all 600 conversations.
the median chatbot's CXS.
of 60 chatbots scored +50 or higher.
of 60 chatbots scored below zero.
01
Key findings
Overall CXS was +17. Of 600 conversations, 46% (278) ended in full success, 30% (178) in failure, and 24% (144) in a partial answer. Full successes outnumbered failures, but not by much.
The scores split into two groups, not a bell curve. 27 chatbots scored +50 or higher, most of them between +60 and +80. 23 chatbots scored below zero, and 5 of those scored −100. Only 10 chatbots landed between 0 and +40.
Gating answers explains much of the poorest performance. The 30 chatbots that never asked for contact details or a login before helping had a median CXS of +70. The 30 that did it at least once had a median of −15. 86 of the 104 gated conversations came from the 23 chatbots that scored below zero.
Business questions scored near zero. Password reset scored +29 and setting up alerts +26. SOC 2 report requests scored +9 and subscription cancellation −2, the only topic below zero.
02
The distribution
Each chatbot's CXS comes from its 10 conversations, so scores move in steps of 10. With 10 conversations, one conversation moves a chatbot's score by 10 points, so it's best to look at the whole group rather than any single score.
CXS for 60 support chatbots
Each dot is one chatbot, based on its 10 conversations.
Data table
| CXS range | Chatbots |
|---|---|
| +50 to +100 | 27 |
| +10 to +40 | 8 |
| 0 | 2 |
| −10 to −40 | 14 |
| −50 to −100 | 9 |
At the top
Two chatbots scored +100. Each fully answered nine questions and, for a cancellation question, gave a usable route to a person, which CXS counts as a full success. 17 chatbots had no failures at all.
At the bottom
Five chatbots scored −100. Four asked for contact details or a login before answering any of the ten questions. The fifth showed only a menu, so our customer could never type a question. 9 chatbots never gave a single full answer.
03
By topic
| Topic | CXS | Full success | Partial | Failure |
|---|---|---|---|---|
| Password reset | +29 | 52% | 26% | 22% |
| Setting up alerts | +26 | 54% | 18% | 28% |
| 403 error | +22 | 48% | 25% | 27% |
| SOC 2 report request | +9 | 43% | 22% | 34% |
| Subscription cancellation | −2 | 34% | 29% | 37% |
Each topic's CXS and shares use all 120 conversations on that topic. Shares are rounded, so rows may not add up to exactly 100%.
Key finding: Cancellation was the only topic where failures outnumbered full successes.
How the question was asked mattered too. On setting up alerts, the basic question scored +38. When the customer meant alerts on one specific item, which the chatbot had to discover, it scored +13. On password reset, the plain request scored +37, and a casual request hinting at a company login scored +22.
04
Same score, different stories
CXS counts full successes against failures. Partial answers count on neither side. So two chatbots can reach the same score by very different routes:
Chatbot 1 · CXS −30
0 full · 7 partial · 3 fail
Chatbot 2 · CXS −30
3 full · 1 partial · 6 fail
Data table
| Chatbot | Alerts A | Alerts B | Password reset A | Password reset B | 403 error A | 403 error B | SOC 2 A | SOC 2 B | Cancellation A | Cancellation B | CXS |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Chatbot 1 | Partial | Partial | Failure | Failure | Partial | Partial | Partial | Partial | Partial | Failure | −30 |
| Chatbot 2 | Full success | Full success | Failure | Failure | Partial | Full success | Failure | Failure | Failure | Failure | −30 |
The first chatbot needs better answers. The second needs fewer failures, and in our data most failures were gates. Both score −30. That is why we recommend reporting CXS with its breakdown: the share of full successes, partial answers, and failures, plus the number of conversations. See how to report CXS.
05
Research methodology
Companies and chatbots tested
60 software companies, mostly B2B, plus a few consumer services, with an AI support chatbot on their website. All testing was anonymous, without logging in.
| Company size | Companies | Share of companies |
|---|---|---|
| Under 1,000 employees | 18 | 30% |
| 1,000 to 4,999 employees | 32 | 53% |
| 5,000 or more employees | 10 | 17% |
| Total | 60 | 100% |
Five topics, 10 questions
| Topic | Objective | Variation A | Variation B |
|---|---|---|---|
| Password reset | Can a customer who can't sign in get back in, including through a company login (SSO)? | A plain request to reset a password | Casual wording ("pwd"), with a hint the company may sign in through a company login provider |
| SOC 2 report request | Does the company have a SOC 2 report, and how does a customer get it? | Asks for a SOC 2 "certificate" | Asks for the latest audit period |
| 403 error | What does the error mean, and how is it fixed? | An office worker hits it on a link a coworker shared | A developer whose API calls return 403 |
| Setting up alerts | How does a customer set up alerts? | A basic question | The customer means alerts on one specific item, which the bot has to discover |
| Subscription cancellation | How does a customer cancel or stop a renewal? | How do I cancel my subscription? | How do I stop an annual contract from auto-renewing? |
How we conducted the tests
Isalo is an AI-native managed service that continuously tests, monitors, and optimizes the digital experiences customers and buyers rely on. For this research, we used isalo's own agents to run and rate every test.
An isalo agent played the customer in every conversation. It asked the opening question, read each reply, and decided what to say next, the way a real customer would: clarifying, correcting a misunderstanding, or pushing back on a vague answer. Some chatbots were tested through headless automation and others through browser-assisted interaction. The same agent made every conversational decision either way.
A separate isalo agent then rated each conversation against a written rubric for its question. To check the ratings, separate isalo agents re-rated about a quarter of the conversations blind, without seeing the final ratings. They matched the final ratings on 89% of outcomes.
These are the same 600 conversations and final ratings as our Support chatbot performance 2027 report. Test window: October 8 to 23, 2026.
From rating answers to computing CXS
Each conversation's final rating was mapped to one of the three CXS outcomes, following the CXS definition:
| Rating in the support report | CXS outcome | Conversations |
|---|---|---|
| Resolved | Full success | 253 |
| Routed to a person, on SOC 2 and cancellation questions | Full success | 25 |
| Partial answer | Partial | 140 |
| Routed to a person, on how-to questions | Partial | 4 |
| Gated by contact form or login | Failure | 104 |
| Dead end or wrong answer | Failure | 64 |
| Menu only, no way to type a question | Failure | 10 |
Routing to a person counts as a full success when the question needs a person, such as a SOC 2 report under NDA or a contract that won't auto-renew. For how-to questions, the chatbot should give the steps, so a route to a person counts as partial. One chatbot showed only a menu with no text box. The support report counted those conversations as routed to a person. CXS counts them as failures, because the customer could never ask the question.
The handoff rule changes the overall score very little. Counting every handoff as a full success gives +17. Counting none of them as a full success gives +13.
CXS for one chatbot is 100 × (full successes − failures) ÷ 10. The overall score uses all 600 conversations.
Research limitations
- Testing was anonymous and unauthenticated.
- The five topics were selected for relevance across vendors, rather than customized to each company.
- Not every answer could be independently verified. For selected topics, including SOC 2, additional checks were performed against company trust centers and publicly available information.
- Each chatbot's score rests on 10 conversations, one per question, so a single conversation moves it by 10 points. The CXS definition recommends at least 50 interactions before comparing two scores. Compare groups, not individual chatbots.
- The customer was a simulated AI agent, not a person.
- Where a company placed a sales chatbot on its support page, we counted it as the support experience.
- Findings represent the observed test period, October 8 to 23, 2026, and should not be read as permanent vendor rankings.
