RESEARCH · CXS · SUPPORT CHATBOTS

We scored 60 support chatbots with CXS™. 27 scored +50 or higher. 23 had a negative score.

What 600 conversations reveal about the state of AI support chatbots, and how to measure yours in 2027.

In October, isalo's AI agents held 600 conversations with the AI support chatbots of 60 software companies, ten questions each, and rated every answer. We have now used the ratings to compute each chatbot's CXS, the Customer Experience Success Index: the share of interactions that end in full success, minus the share that end in failure, on a scale from −100 to +100.

The result is not a bell curve. Most chatbots either did well or did badly. Few landed in the middle.

60 chatbots · 600 conversations · tested October 8 to 23, 2026 · CXS version 1.0

+17

CXS across all 600 conversations.

+35

the median chatbot's CXS.

27

of 60 chatbots scored +50 or higher.

23

of 60 chatbots scored below zero.

01

Key findings

Overall CXS was +17. Of 600 conversations, 46% (278) ended in full success, 30% (178) in failure, and 24% (144) in a partial answer. Full successes outnumbered failures, but not by much.

The scores split into two groups, not a bell curve. 27 chatbots scored +50 or higher, most of them between +60 and +80. 23 chatbots scored below zero, and 5 of those scored −100. Only 10 chatbots landed between 0 and +40.

Gating answers explains much of the poorest performance. The 30 chatbots that never asked for contact details or a login before helping had a median CXS of +70. The 30 that did it at least once had a median of −15. 86 of the 104 gated conversations came from the 23 chatbots that scored below zero.

Business questions scored near zero. Password reset scored +29 and setting up alerts +26. SOC 2 report requests scored +9 and subscription cancellation −2, the only topic below zero.

02

The distribution

Each chatbot's CXS comes from its 10 conversations, so scores move in steps of 10. With 10 conversations, one conversation moves a chatbot's score by 10 points, so it's best to look at the whole group rather than any single score.

CXS for 60 support chatbots

Key finding: Half the chatbots scored +35 or higher, but the group splits in two. A large cluster scored between +60 and +80, and a long tail ran down to −100.
Data table
Distribution of chatbot CXS scores
CXS rangeChatbots
+50 to +10027
+10 to +408
02
−10 to −4014
−50 to −1009

At the top

Two chatbots scored +100. Each fully answered nine questions and, for a cancellation question, gave a usable route to a person, which CXS counts as a full success. 17 chatbots had no failures at all.

At the bottom

Five chatbots scored −100. Four asked for contact details or a login before answering any of the ten questions. The fifth showed only a menu, so our customer could never type a question. 9 chatbots never gave a single full answer.

03

By topic

CXS works as one overall index, and it breaks down as far as you need. Here it is by topic. Each topic's CXS is computed from all 120 conversations on that topic: 60 chatbots, two questions each.

CXS and outcomes by topic
TopicCXSFull successPartialFailure
Password reset+2952%26%22%
Setting up alerts+2654%18%28%
403 error+2248%25%27%
SOC 2 report request+943%22%34%
Subscription cancellation−234%29%37%

Each topic's CXS and shares use all 120 conversations on that topic. Shares are rounded, so rows may not add up to exactly 100%.

Key finding: Cancellation was the only topic where failures outnumbered full successes.

How the question was asked mattered too. On setting up alerts, the basic question scored +38. When the customer meant alerts on one specific item, which the chatbot had to discover, it scored +13. On password reset, the plain request scored +37, and a casual request hinting at a company login scored +22.

04

Same score, different stories

CXS counts full successes against failures. Partial answers count on neither side. So two chatbots can reach the same score by very different routes:

Chatbots 1 and 2: both −30
−1000+100
Two of the 60 chatbots scored −30. Here is how each got there.
Alerts
Password
403 error
SOC 2
Cancel
ABABABABAB

Chatbot 1 · CXS −30

0 full · 7 partial · 3 fail

Chatbot 2 · CXS −30

3 full · 1 partial · 6 fail

Full successPartial successFailure
Data table
Outcomes for two chatbots with CXS −30
ChatbotAlerts AAlerts BPassword reset APassword reset B403 error A403 error BSOC 2 ASOC 2 BCancellation ACancellation BCXS
Chatbot 1PartialPartialFailureFailurePartialPartialPartialPartialPartialFailure−30
Chatbot 2Full successFull successFailureFailurePartialFull successFailureFailureFailureFailure−30

The first chatbot needs better answers. The second needs fewer failures, and in our data most failures were gates. Both score −30. That is why we recommend reporting CXS with its breakdown: the share of full successes, partial answers, and failures, plus the number of conversations. See how to report CXS.

05

Research methodology

Companies and chatbots tested

60 software companies, mostly B2B, plus a few consumer services, with an AI support chatbot on their website. All testing was anonymous, without logging in.

Company size, by employee count
Company sizeCompaniesShare of companies
Under 1,000 employees1830%
1,000 to 4,999 employees3253%
5,000 or more employees1017%
Total60100%

Five topics, 10 questions

Five topics, 10 questions
TopicObjectiveVariation AVariation B
Password resetCan a customer who can't sign in get back in, including through a company login (SSO)?A plain request to reset a passwordCasual wording ("pwd"), with a hint the company may sign in through a company login provider
SOC 2 report requestDoes the company have a SOC 2 report, and how does a customer get it?Asks for a SOC 2 "certificate"Asks for the latest audit period
403 errorWhat does the error mean, and how is it fixed?An office worker hits it on a link a coworker sharedA developer whose API calls return 403
Setting up alertsHow does a customer set up alerts?A basic questionThe customer means alerts on one specific item, which the bot has to discover
Subscription cancellationHow does a customer cancel or stop a renewal?How do I cancel my subscription?How do I stop an annual contract from auto-renewing?

How we conducted the tests

Isalo is an AI-native managed service that continuously tests, monitors, and optimizes the digital experiences customers and buyers rely on. For this research, we used isalo's own agents to run and rate every test.

An isalo agent played the customer in every conversation. It asked the opening question, read each reply, and decided what to say next, the way a real customer would: clarifying, correcting a misunderstanding, or pushing back on a vague answer. Some chatbots were tested through headless automation and others through browser-assisted interaction. The same agent made every conversational decision either way.

A separate isalo agent then rated each conversation against a written rubric for its question. To check the ratings, separate isalo agents re-rated about a quarter of the conversations blind, without seeing the final ratings. They matched the final ratings on 89% of outcomes.

These are the same 600 conversations and final ratings as our Support chatbot performance 2027 report. Test window: October 8 to 23, 2026.

From rating answers to computing CXS

Each conversation's final rating was mapped to one of the three CXS outcomes, following the CXS definition:

From support report ratings to CXS outcomes
Rating in the support reportCXS outcomeConversations
ResolvedFull success253
Routed to a person, on SOC 2 and cancellation questionsFull success25
Partial answerPartial140
Routed to a person, on how-to questionsPartial4
Gated by contact form or loginFailure104
Dead end or wrong answerFailure64
Menu only, no way to type a questionFailure10

Routing to a person counts as a full success when the question needs a person, such as a SOC 2 report under NDA or a contract that won't auto-renew. For how-to questions, the chatbot should give the steps, so a route to a person counts as partial. One chatbot showed only a menu with no text box. The support report counted those conversations as routed to a person. CXS counts them as failures, because the customer could never ask the question.

The handoff rule changes the overall score very little. Counting every handoff as a full success gives +17. Counting none of them as a full success gives +13.

CXS for one chatbot is 100 × (full successes − failures) ÷ 10. The overall score uses all 600 conversations.

Research limitations

  • Testing was anonymous and unauthenticated.
  • The five topics were selected for relevance across vendors, rather than customized to each company.
  • Not every answer could be independently verified. For selected topics, including SOC 2, additional checks were performed against company trust centers and publicly available information.
  • Each chatbot's score rests on 10 conversations, one per question, so a single conversation moves it by 10 points. The CXS definition recommends at least 50 interactions before comparing two scores. Compare groups, not individual chatbots.
  • The customer was a simulated AI agent, not a person.
  • Where a company placed a sales chatbot on its support page, we counted it as the support experience.
  • Findings represent the observed test period, October 8 to 23, 2026, and should not be read as permanent vendor rankings.

Want to know your CXS?

isalo's AI agents can test your support chatbot, AI assistants, and help content against the questions your customers ask, rate every answer, compute your CXS, and track it over time.

What is CXS?

License and citation

This research is licensed under Creative Commons Attribution 4.0 (CC BY 4.0). You may share and reuse the findings with attribution to isalo.

Suggested citation: isalo. CXS for 60 support chatbots. November 18, 2026. https://isalo.ai/research/support-chatbot-cxs-2027

CXS is a trademark of A6 Group LLC.