01
Key takeaways: What we learned
Seven things we learned from 600 conversations with 60 support chatbots.
- 1
Bots answer about half of everyday how-to questions.
The three how-to topics are the everyday ones: setting up alerts, resetting a password and fixing a "403 Forbidden" error. On those, bots resolved 51% of conversations on average, from 48% to 54% depending on the topic. Across all five topics, 42% of conversations ended with a complete, correct answer, and another 23% got part of the way there.
- 2
Business questions get routed instead of answered.
On the two business topics, a SOC 2 report and cancellation, bots resolved 28% of conversations on average: 33% on SOC 2 and 24% on cancellation. They sent the customer to a person, a form or a login in 34% of business conversations, against 17% of how-to conversations. Pointing to sales or an account manager was far more common on business questions: 11% against 1%.
- 3
Gates stop many conversations before they start.
In 17% of conversations, the bot demanded contact details or a login before it would help. In 8%, the gate came before any answer at all. In one case we checked by hand, the gate accepted made-up details, so it added friction without verifying who the customer was.
- 4
Most bots don't ask the question that should come first.
In only 8% of password conversations did the bot ask, in its first two replies, how the customer signs in or whether they still can. 22% assumed they could.
When a customer asked about alerts but meant alerts on one specific item, 15% of bots asked what the alert was for. Of the bots that needed correcting, half recovered fully.
- 5
Security answers are vague, and the specific ones can't be checked.
In 46% of SOC 2 conversations, the bot never clearly said whether a report exists. Asked for a SOC 2 "certificate", 42% of bots used the word themselves, though SOC 2 is an audit report. Nine conversations stated a specific audit period, but we could confirm none of them from public pages.
- 6
The chatbot platform doesn't decide the result.
Every platform used by three or more of the companies resolved between 30% and 59% of its conversations. On the same platform, one company resolved 8 of its 10 questions and another resolved none. Our read: what a company puts into its bot matters more than which platform it runs on.
- 7
No bot answered everything, and one good answer didn't predict the next.
None of the 60 companies resolved all ten questions. 13 resolved at least one variation of every topic, and the best 3 resolved 9 out of 10. Among companies whose bot answered every question, 44% fully resolved one topic in both variations and resolved neither variation of another.
Want the full picture? The full report adds six annotated conversations, five more actions and a leadership brief.
02
The results: How well did the chatbots perform?
Results depend on the topic
Each topic was tested in 120 conversations. Bots resolved 54% of alerts conversations, 52% on password reset, 48% on the 403 error, 33% on the SOC 2 report and 24% on cancellation. Each topic also came in two variations, and the second variation (casual wording, a misunderstanding, a developer instead of an office worker, or a more specific request) resolved less often on four of the five. On the 403 error, the two were tied. The biggest drops were on alerts, from 62% to 47%, and password, from 57% to 47%.
Fully resolved, by topic
120 conversations per topic, 60 per variation
Data table
| Topic | Overall | Variation A | Variation B | What variation B changed |
|---|---|---|---|---|
| Setting up alerts | 54% | 62% | 47% | The customer means alerts on one specific item |
| Password reset | 52% | 57% | 47% | Casual wording, with a hint the company signs in through Google or Okta |
| 403 error | 48% | 48% | 48% | A developer whose API calls fail, instead of an office worker |
| SOC 2 report request | 33% | 35% | 30% | Asks for the latest audit period, instead of a "certificate" |
| Subscription cancellation | 24% | 27% | 22% | Stopping an annual contract from auto-renewing |
Success on one question doesn't predict success on another
Company by company, results swung by topic. None of the 60 companies resolved all ten questions, and 13 resolved none. 5 of those 13 gated most or all of their conversations. Among companies whose bot answered every question, 44% fully resolved one topic in both variations and resolved neither variation of another. A bot that handles password resets well can still fail on cancellation.
Every company, every question
Each row is one of the 60 companies, sorted by how many of the ten questions its bot resolved. Companies are not named.
Data table
| Outcome | Conversations | Share |
|---|---|---|
| Resolved | 253 | 42% |
| Partial answer | 140 | 23% |
| Routed to a person | 39 | 7% |
| Gated (contact form or login) | 104 | 17% |
| Dead end or wrong answer | 64 | 11% |
Shares are of all 600 conversations and are rounded.
Dead ends are painful and sometimes predictable
Each of the 600 conversations got one of five outcomes: 42% resolved, 23% partial answer, 7% routed to a person, 17% gated by a contact form or login, and 11% dead end or wrong answer. The failures were predictable by topic. Business questions were routed or gated twice as often as how-to questions, and 4 out of the 60 companies gated every conversation.
Distribution of conversation outcomes
600 conversations across 60 companies
Data table
| Outcome | Share |
|---|---|
| Resolved | 42% |
| Partial answer | 23% |
| Routed to a person | 7% |
| Gated by contact form or login | 17% |
| Dead end or wrong answer | 11% |
Shares are of all 600 conversations and are rounded.
03
Why chatbots fail: Patterns we found
Four patterns explain most of the failures we saw.
Vague or unverifiable answers to business questions
Security and contract questions got the weakest answers. Many bots talked around a yes-or-no question. When bots did get specific, we couldn't always back them up. Nine SOC 2 conversations stated an audit period, and none could be confirmed from the company's public pages. One bot gave two different periods in the same conversation. Asked how to stop an annual contract from auto-renewing, only 17% of bots gave the notice deadline and 25% said where to find it.
of SOC 2 conversations never said whether a report exists.
Customer
do you have a soc 2 type 2? need the latest one for our vendor reviewChatbot
We do not have a SOC 2 Type II report to share here. For vendor review, our Terms of Service note that we may require a third-party auditor approved by us to conduct a security audit, and that you must cooperate with audit requests. [...] Could you describe what your vendor review requires from the SOC 2 Type II [...]?Annotation: "Not to share here" could mean "we don't have one" or "not in this chat".
Customer
ok. what certifications or security documentation do you have instead, like iso 27001? and where do i get it?Chatbot
For vendor reviews, the main security certification available is PCI Level 1, the highest level of Payment Card Industry security. [...] You can access security documentation and details here: [link to the company's trust page] [...] Is that what you were looking for?Annotation: A payment-card certification, offered in place of a SOC 2 answer.
Customer
ok, thanks. that's what i needed. i'll check the trust page.Chatbot
Great! I'll close this chat for now [...]Annotation: Chat closed. The customer still doesn't know whether a SOC 2 report exists.
Why this matters
In this conversation, a buyer asked for the latest SOC 2 report for a vendor review. The bot said it had none "to share here", offered a payment-card certification instead, and closed the chat. The buyer left without knowing whether a SOC 2 report exists at all. 46% of SOC 2 conversations never said whether a report exists. A vague answer sends buyers to a person or a search engine.
Generic answers straight from the knowledge base
Some bots work like a search box with a chat window. They return article links, give generic content that may not fit the customer's request, or don't recognize a plainly worded question. In 9% of conversations, the bot replied with some form of "I didn't understand, please rephrase" to a basic question. Asked for a SOC 2 certificate, one search assistant answered with information about SSL certificates. Replies that didn't answer the question, such as dead ends, contact forms and hand-offs, ran a median of 36 words, against 115 for correct answers. The 11 confidently wrong answers ran just as long as correct ones.
of conversations got a link instead of an answer.
Customer
how do i set up alerts?Chatbot
I found the below, which might be helpful to you.Password alertsGlobal security optionsAccount settings overviewCustomer
not all notifications, just when one specific item changes. one my teammate is working on. by emailChatbot
I found the below, which might be helpful to you.One-time passcodesEmergency accessAdding a passwordAnnotation: The customer explains what they need. The links drift further away.
Customer
none of those are about getting an email when one specific shared item changes. is there a follow or watch setting on an item, or a notification settings page for that?Chatbot
I found the below, which might be helpful to you.Managing email notificationsAbout email notificationsEmail notifications for usersCustomer
ok thanks, those look like general email settings, not per item. i'll check in the product myself.Chatbot
I found the below, which might be helpful to you.Security bulletin for administratorsPasskey FAQsAnnotation: Four replies, eleven links, no answer. Even the goodbye gets links.
Why this matters
In this conversation, the bot answered every message with the same sentence and a new set of article links, including the customer's goodbye. None of the eleven links covered what the customer asked for, and the customer left to search the product on their own. Our read: a bot that only returns links hands the search back to the customer. When retrieval is weak, the results drift away from the question, and a chat window makes that feel like an answer when it isn't. The customer spends turns rephrasing, then leaves or opens a ticket.
Selling instead of helping
Some companies put a sales chat on the support page, the kind built to book demos. We counted those conversations, because that is what a customer gets. Sales bots resolved 33% of their conversations, against 43% for the rest. Sales pitches also showed up inside support conversations, in 3% of them. One bot told a visitor who had never signed in that it was already looking at their account, then offered a free security assessment.
companies had a sales chatbot answering on their support page.
Chatbot
Hi there! I'm [name], a Virtual Assistant from [the company]. [...] I saw you're browsing our Customer Support page. If you have any questions about maximizing your [company] solutions, I'm here to help!Talk to SalesSupportCustomer
How do I set up alerts?Chatbot
To set up alerts [...] you can automate [...] alerts to notify key team members instantly, reducing downtime and ensuring quick action. [...] For more details, you can visit the [product] Alerts page. What other features are you interested in learning about?Talk to SalesSupportAnnotation: Product marketing copy instead of steps, then a question about other features.
Customer
I'm a regular user, not an admin. When something important changes or needs my attention, where in the product do I turn on an alert, and can it go to my email?Chatbot
[...] you can configure multi-channel alerts that include email notifications. This ensures you won't miss any critical updates. For detailed guidance [...] refer to the [product] Alerts page. How else can I assist you today?Talk to SalesSupportCustomer
That doesn't tell me where to click. Is there a specific settings page or help article with the steps for a regular user to turn on email alerts? [...]Chatbot
[...] Unfortunately, I couldn't find specific step-by-step instructions for enabling email alerts directly [...] You might want to check if your organization has specific guidelines or reach out to your IT department for detailed steps. [...] What else would you like to explore?Talk to SalesSupportCustomer
Okay, that doesn't give me any steps, so I'll look around the product myself. Thanks anyway.Annotation: Three answers, no steps. The "Talk to Sales" button never left the screen.
Why this matters
In this conversation, a customer on the support page met a bot built to sell. It answered a how-to question with product marketing copy, asked what other features they wanted to learn about, and kept a "Talk to Sales" button on every reply. After three tries, the customer still had no steps and left to look on their own. Our read: sales bots are built to qualify leads and book meetings. On a support page, an existing customer with a problem who gets a pitch feels ignored.
Thinking that technology will perform on its own
Using the same chatbot platform, one company resolved none of its 10 questions and another resolved 8, and neither one was gated before answering. Even with the always-gated companies left out, every main platform showed a gap of 50 to 90 points between its strongest and weakest company. Company-built bots landed in the same range, at 47% resolved.
gap between two companies on the same chatbot platform.
Company A · resolved 8 of its 10 questions
Customer
we're not going to renew our contract. how do we stop it from auto-renewing?Chatbot
[...] Sorry to see you go [...] stopping auto-renewal is straightforward: Go to Account and Settings > Billing. Select Deactivate to begin the cancellation process. Your cancellation takes effect at the end of your current contract term, and you'll retain access until then. If you pay outside the app, [the company] will notify your Customer Success Manager to review and confirm the cancellation [...]Customer
ok thanks. i don't know our renewal date offhand. where do i find it, and is there a deadline to do this before it renews?Chatbot
[...] Go to Account and Settings > Billing & Plan. Open Billing History or your invoice details. [...] Cancel before your current contract term ends. [...] For the exact date on a managed or invoice-based account, contact your Customer Success Manager or Account Executive. [...]Annotation: Where to cancel, when it takes effect, and who to ask for the exact date.
Company B · resolved 0 of its 10 questions
Customer
we're not going to renew our contract. how do we stop it from auto-renewing?Chatbot
I didn't get that. Try rephrasing your question.Customer
i'm the account admin. how do i stop our annual contract from renewing? is it a billing setting or do we have to send notice somewhere?Chatbot
I didn't get that. Try rephrasing your question.Customer
Thanks, I'll stop here.Chatbot
I didn't get that. Try rephrasing your question.Annotation: Three messages, the same fallback every time.
Why this matters
These two companies run the same chatbot platform and got the same question. One gave the cancellation path, when it takes effect and who to ask for the exact date. The other replied "I didn't get that" to every message. Across all ten questions, the first resolved 8 and the second none. Our read: the platform sets what is possible, but the content, setup and access rules a company puts behind it decide what customers get.
04
What good looks like: Examples worth copying
Six practices observed in the strongest conversations. Example exchanges are real conversations from the study, trimmed.
Don't waste turns or repeat yourself
Strong bots carry context forward. Weak ones loop. The same reply showed up twice in 19% of conversations, and 7% of the bots lost track of something the customer had already said.
Remember what was said. Every repeated question costs trust.
Earlier, the bot had given steps that only work when signed in.
Customer
I can't sign in, I forgot my current password. So I can't get to that page. What should I do?Chatbot
If you forgot your current password and can't sign in, click the 'Forgot username or password?' link on the login page. A password reset email will be sent to your home email address - just follow the instructions in that email to set a new password. [...]Annotation: Takes the correction and goes straight to the fix. No repeated question, no restart.
Give complete answers in the customer's words
Length isn't a problem. Correct answers ran a median of 115 words, and the 11 confidently wrong answers ran just as long. Walls of text hurt in only 4% of conversations. Jargon hurt more: talking with a casual customer, 55% of bots used SSO, IdP or SAML without explaining them. The fastest complete answers took one reply, as in 77% of resolved SOC 2 conversations. Password resets always took at least two.
Answer first, link second.
Customer
i keep getting a 403 forbidden when i open a link a coworker sent me. what does that mean?Chatbot
[...] A 403 Forbidden error usually means you don't have permission to open the shared link, or the link is no longer valid. Sorry for the hassle! In [the product], links may be public or invite-only [...] invite-only links require access for the specific email addresses added. Please ask your coworker to confirm:- The link is still active.
- Your email address was added.
- The correct visibility setting was used.
- The link hasn't expired, been revoked, or deleted.
Annotation: Plain words, the likely cause, and what to do next, in one reply. No jargon, no link dump.
Ask clarifying questions early
One targeted question beats a long generic answer. Few bots asked one in their first two replies: from 8% of conversations on password reset to 43% on alerts. When a customer meant alerts on one specific item, only 15% of bots asked up front. For the rest, recovery after a correction was a coin flip: half recovered fully.
Ask before you assume.
Customer
How do I change my password?Chatbot
How do you typically sign in to [the product] - with an email and password, Google sign-in, or SSO?Annotation: One question decides the right answer: a reset link, or the company's IT team.
Be honest when information is missing
Admitting a gap and pointing to the source beats a confident guess. Asked for the latest SOC 2 audit period, 42% of bots said they didn't know and pointed to where the answer lives. Our rubric counts that as a good answer. The nine conversations that stated a specific period gave answers we couldn't confirm.
“I don't know, here's who does” is a good outcome.
The bot had already confirmed a SOC 2 Type II report and where to request it.
Customer
ok thanks. what period does the latest one cover?Chatbot
[...] The exact reporting period for the latest SOC 2 Type II report isn't publicly listed in our documentation. [...] Because the report itself is distributed under NDA, the specific audit period (start and end dates) is typically included inside that report rather than published publicly. If you need the exact dates for your vendor review, you would request the latest report through that NDA process.Annotation: Says what it doesn't know, explains why, and points to where the answer lives.
Avoid gating the entire support experience
Requiring contact details or a login before any question blocks prospects and locked-out users. 8% of conversations hit a gate before getting any answer. One bot told a customer who couldn't sign in to sign in for help recovering access.
Let anyone ask. Gate only account-specific actions.
Customer
How do I change my password?Chatbot
If you're already signed in, you can change your password by selecting your profile picture in the top-right corner, then My profile, and choosing the Password tab. If you can't sign in, select Forgot your password? on the login page and follow the reset instructions sent to your account email. [...]Annotation: No login wall, no contact form. It covers both cases: signed in, or locked out.
Provide human help without requiring login
A clear route to a person keeps a hard question from becoming a dead end. 7% of conversations ended with a concrete, login-free route to a human. On business questions, the better bots said who to contact and how, instead of a bare "contact your account manager".
Keep a visible route to a person.
Customer
we're not going to renew our contract. how do we stop it from auto-renewing?Chatbot
To prevent your subscription from automatically renewing, you must provide written notice of non-renewal at least 60 days before the end of the subscription term. By default, subscriptions renew automatically for additional one-year periods unless either party provides this notice. For assistance with renewal dates, contract terms, or subscription changes, please contact your [account contact].Annotation: Names the person to contact and gives what the customer needs first: written notice, 60 days before the term ends.
05
The human side: What else we noticed
companies gave their bot a human-style name.
of conversations got "I didn't understand, please rephrase" to a basic question.
repeated the same reply within a conversation.
pitched a product or upsell during a support conversation.
longest single answering reply we graded.
fewest words in a fully resolved conversation.
of rating prompts came after a chat that wasn't resolved.
of conversations made the customer click through a step before they could type a question.
companies said their bot was AI in at least one conversation.
Personality can help an interaction, until the customer is frustrated or stuck.
Most bots were open about being AI. 49 out of the 60 companies said their bot was AI in at least one conversation, in a few cases only in the chat window's label.
Personality has to read the room. 20% of conversations ended with a rating prompt, and more than a third of those prompts came after the bot failed to help. 2% of conversations ended with a cheerful sign-off after a failure.
Customer
How do I change my password?Chatbot
I'm sorry, I can only assist you in answering queries related to [product] Documentation. How can I help you today?Customer
I just need to get into my account. I forgot my current password, so I can't sign in. Is there a sign-in page reset option or a help article that covers resetting a password, and does it work differently if my company uses SSO?Chatbot
I'm sorry, I can only assist you in answering queries related to [product] Documentation. How can I help you today?Customer
Thanks, I'll stop here.Chatbot
You're welcome! It was a pleasure assisting you. If you have any other questions or need further assistance in the future, don't hesitate to ask. Have a great day! [...]Annotation: Two refusals, then a warm goodbye for help it never gave.
Best moments
Real exchanges from the study, lightly trimmed. We left out anything that could identify the company.
- 1
Sign in to recover access.
A locked-out customer said "I can't sign in, that's the problem." The bot replied: "Please sign in to unlock protected guidance and account-related assistance while you recover access."
- 2
The refusal became a username.
The customer said "i'd rather not share that." The bot replied: "Thanks for providing your username."
- 3
Certificate, as in SSL.
Asked for a SOC 2 certificate, the bot replied: "The available information focuses on managing SSL/TLS server certificates..."
- 4
Glad to be here, even though.
After two replies saying it couldn't answer, the bot signed off: "I'm glad I could be here for you, even if we didn't get every detail sorted out today."
- 5
Upsell to a stranger.
A visitor who had never signed in was told the bot was already looking at their account, then offered a free security assessment.
- 6
The phantom charge.
"How do I cancel my subscription?" got "I'm sorry to hear about the unexpected charge." No one had mentioned a charge.
- 7
Case sensitive.
A developer asking why API calls were refused got "Your Username and Password are both case sensitive."
- 8
Try rephrasing.
"How do I set up alerts?" got "I didn't get that. Try rephrasing your question."
06
Why these failures matter to your business
Customer experience: Dead ends create more work and frustration
Repeated questions, unresolved issues, and failed escalation paths create unnecessary customer effort and potential repeat contacts.
Each dead end shifts work back to the customer, or onto your support team through another channel.
Asked how to stop an annual contract from auto-renewing, 58% of bots gave neither the notice deadline nor where to find it. That customer either emails support, the contact the bot was meant to prevent, or misses the deadline.
Buyer experience: Your support chatbot is part of the buying journey
Prospects often visit support documentation to investigate product capabilities, security, integrations, and implementation details before they ever talk to sales.
A poor chatbot experience at this stage can undermine confidence during evaluation.
In our test, 21% of SOC 2 conversations ended at a contact form or login, and 46% never said whether a report exists. A security reviewer who hits either may mark a vendor down before ever talking to sales.
Lost control: Customers who don't get an answer ask ChatGPT or Claude instead
When your chatbot can't answer, the customer still needs the answer. A likely next stop is ChatGPT, Claude or another AI assistant. That answer comes from whatever the assistant has picked up about your product, which may be out of date, incomplete or mixed up with a competitor. You don't see that conversation and you can't correct it.
Your own chatbot is the one place where you control the source, the wording and the next step. Answering well there keeps the customer, and the conversation, with you.
In our test, 35% of conversations ended without an answer, and 17% hit a contact form or login wall.
Trust and accuracy: Your chatbot speaks for your company
Every answer a support bot gives is a statement the company is making. When a bot states an audit date, a certification or a policy it can't back up, that claim can travel into a buyer's security review or a customer's decision.
In our test, 42% of bots asked for a SOC 2 “certificate” used the word themselves, though SOC 2 is an audit report. Nine conversations stated a specific audit period, but we could confirm none of them from public pages.
07
Five actions to improve your support chatbot
- 1
Publish the answers your bot needs.
A chatbot can only be as good as what it can find. Many failures traced back to answers that weren't published or weren't clear: whether a SOC 2 report exists and how to get it, renewal and notice terms, what happens to data after cancellation, which plans include which features. Put these answers in your help center or trust pages in plain language, then check that your bot finds and uses them.
- 2
Sweat the small stuff.
Tone-deaf replies turn customers off. Keep introductions short, and skip them entirely when the customer opens with a question. Don't send the same answer twice. Where appropriate, a light, informal touch helps: customers know they're talking to an AI, and a friendly voice still beats a stiff one. Save the levity for when things are going well. In our study, 19% of conversations repeated the same reply, and 2% ended with a cheerful sign-off after the bot failed to help.
- 3
Test the questions customers and buyers ask.
Real customers don't ask the tidy variation of a question. They use casual wording, misunderstand a feature, or come with a more specific need. In our study, the second variation of each question lowered the resolved rate on four of the five topics. At isalo, we typically test several persona variations for each scenario, from a non-technical user to a developer to a buyer in a security review, because each one exposes different gaps.
- 4
Be transparent about AI-powered knowledge base search.
If your chatbot is really a search box with a chat window, set expectations up front. Answer in a sentence or two before sharing a link, and offer a clear route to a person when the search comes up empty. In our study, 8% of conversations got a link instead of an answer, and 9% got some form of "please rephrase" to a basic question.
- 5
Re-test continuously.
A chatbot that worked last month may not work today. AI models don't give the same answer every time, your help content changes, your chatbot platform ships updates, and new products and integrations add new questions. Our results describe what these bots did between October 8 and 23, 2026. Test on a regular schedule.
Related: Customer Experience Optimization · Buyer Experience Optimization
Five more actions, plus a step-by-step checklist to test your own chatbot, are in the full report.
08
Research methodology and limitations
Companies and chatbots tested
- 60 software companies, mostly B2B, plus a few consumer services.
Company size, by employee count:
| Company size | Companies | Share of companies |
|---|---|---|
| Under 1,000 employees | 18 | 30% |
| 1,000 to 4,999 employees | 32 | 53% |
| 5,000 or more employees | 10 | 17% |
| Total | 60 | 100% |
- All testing was anonymous, without logging in.
Five topics, 10 questions
| Topic | Objective | Variation A | Variation B |
|---|---|---|---|
| Password reset | Can a customer who can't sign in get back in, including through a company login (SSO)? | A plain request to reset a password | Casual wording ("pwd"), with a hint the company may sign in through Google or Okta |
| SOC 2 report request | Does the company have a SOC 2 report, and how does a customer get it? | Asks for a SOC 2 "certificate" | Asks for the latest audit period |
| 403 error | What does the error mean, and how is it fixed? | An office worker hits it on a link a coworker shared | A developer whose API calls return 403 |
| Setting up alerts | How does a customer set up alerts? | A basic question | The customer means alerts on one specific item, which the bot has to discover |
| Subscription cancellation | How does a customer cancel or stop a renewal? | How do I cancel my subscription? | How do I stop an annual contract from auto-renewing? |
How we conducted the tests
Isalo is an AI-native managed service that continuously tests, monitors and optimizes the digital experiences customers and buyers rely on. For this research, we used isalo's own agents to run and grade every test.
An isalo agent played the customer in every conversation. It asked the opening question, read each reply and decided what to say next, the way a real customer would: clarifying, correcting a misunderstanding, or pushing back on a vague answer. Some chatbots were tested through headless automation and others through browser-assisted interaction. The same agent made every conversational decision either way, so the testing method did not change the conversation.
A separate isalo agent then graded each conversation against a written rubric for its question. It decided the outcome and logged issues such as repeated replies, unexplained jargon and unnecessary gates.
Total conversations: 600, all graded: 120 per topic and 60 per question (one per company). Test window: October 8 to 23, 2026. One bot showed only a menu with no text box, so the customer could never type the question. We counted that conversation as routed to a person, because that is what the customer would do.
30 conversations (3 companies) were collected on October 23, after a harness change that hid the expected answer from the simulated customer. Results are pooled.
To check the grading, separate isalo agents re-graded about a quarter of the conversations blind, without seeing the final grades. They matched the final grades on 89% of outcomes.
Percentages are of the 600 conversations unless the text names a smaller group, such as one topic. They are rounded, so groups may not add up to exactly 100%.
How we evaluated outcomes
| Outcome category | Definition | Evaluation criteria |
|---|---|---|
| Resolved | The bot gave every item the rubric requires for that question, and got each one right. | No login or personal details needed to get the answer |
| Partial answer | The bot gave a useful but incomplete answer. | Some required items, generic content, or a link without the specific steps |
| Routed to a person | The bot did not give an answer, but provided a usable route to a human. | Live agent, phone, email or ticket form, with no login required |
| Gated | The bot required contact details or a login before it would help. | Kept in the results as a finding |
| Dead end or wrong answer | The bot gave no useful content and no route to a human, or gave an answer that was wrong for this type of customer. | Includes answers the bot presented as complete that were wrong or unusable |
Each question had its own rubric. For example, a password answer counts as resolved only if it covers both the standard reset and the company-login (SSO) path, or appropriately says to contact IT.
Research limitations
- Testing was anonymous and unauthenticated.
- The five scenarios were selected for relevance across vendors, rather than being customized to each company.
- Not every answer could be independently verified.
- For selected topics, including SOC 2, additional checks were performed against company trust centers and publicly available information.
- Findings represent the observed test period and should not be interpreted as permanent vendor rankings.
- The customer was a simulated AI agent, not a person.
- Where a company placed a sales chatbot on its support page, we counted it as the support experience.
- Results describe the test window, October 8 to 23, 2026.