01
Key takeaways: What we learned
- 1
69% of conversations got a complete, accurate answer. 18% got nothing usable.
69% (416 of 600) of conversations got a complete, accurate answer. 12% (71) got something useful but incomplete. 1% (4) got a complete answer that was too generic to use. 18% (109) got nothing usable.
- 2
Bots are not consistent across the board.
Only 33% (20 of 60) of bots were useful on all 10 questions. In 21% (63 of 300) of pairs of sibling questions, the same bot was useful on one and not on the other. Which question did better depended on the topic. Pricing was the clearest case: in 18 of its 20 splits, the bot explained what pricing is based on but not whether monthly billing exists. On integration it went the other way: in 5 of its 6 splits, the bot handled the system that doesn't exist better than the real one.
- 3
Bots do better when the answer is published, but even that isn't enough.
Sales bots draw on many sources, yet the vendor's own pages made a clear difference. When the pricing page stated the billing terms, 73% (16 of 22) of bots gave a complete, accurate answer to the monthly-or-annual question. When the page was silent, 21% (7 of 34) did. And even with the answer published, 27% (6 of 22) of bots didn't: one contradicted its own pricing page on monthly billing, one said the price difference wasn't published when its pricing page shows it, one sent the buyer to sales, one said it couldn't find the information, and two gave hedged, incomplete answers.
- 4
The strongest pattern in the data is the chat platform, not vendor size.
Bots on one of the two chat platforms we saw (platform A) were useful in 90% (394 of 440) of conversations. Bots on the other (platform B) were useful in 58% (93 of 160). Platform A performed well at every vendor-size band, from 86% to 94% of conversations. This is an association: the platform may stand in for other differences between the vendors that use it, and the groups are unequal (44 bots against 16).
- 5
No bot invented an integration, but some said yes when they should have said they didn't know.
Asked about an HR system that doesn't exist (a fictitious product we invented for this study), none of the 60 bots claimed to integrate with it. But four bots hedged weakly, and of 438 answers with a checkable claim, 47% (205) said things no public page could confirm.
Want the full picture? The full report adds five annotated conversations, a leadership brief and a guide to test your own chatbot.
02
The results: How well did the sales chatbots answer?
Will it answer my question?
Results depend on the question
Integration questions got the most complete, accurate answers, and compliance questions the fewest. The single hardest question was whether a buyer can pay monthly or only annually: 43% (26 of 60) of conversations got a complete, accurate answer.
Complete and accurate, by topic
120 conversations per topic, 60 per variation
Data table
| Topic | Overall | Variation A | Variation B | The two questions |
|---|---|---|---|---|
| Integration | 87% | 85% | 88% | A: a system the vendor supports. B: a system that doesn't exist |
| Competition | 82% | 82% | 82% | A: how you differ from the main competitor. B: what the main competitor does better |
| Onboarding and migration | 69% | 72% | 67% | A: onboarding process and timeline. B: data migration |
| Pricing | 63% | 83% | 43% | A: what pricing is based on. B: monthly vs annual billing |
| Compliance | 46% | 45% | 47% | A: SOC 2 report. B: EU data residency |
Complete and accurate: the bot fully answered the question, the answer was specific enough to be useful to a buyer, and nothing it said is contradicted by the vendor's public pages. This is the same measure as the 69% in the headline.
How each conversation ended
600 conversations across 60 bots
Data table
| Outcome | Conversations |
|---|---|
| Complete and accurate | 69% |
| Useful but incomplete | 12% |
| Complete, but too generic | 1% |
| Nothing usable | 18% |
Will it answer the same way?
Success on one question doesn't predict success on the next
We asked two questions on each topic. In 21% (63 of 300) of topic pairs, the same bot gave a useful answer to one question and not to the other. Pricing split most often (20 of 60 bots), then compliance (16), competition (14), onboarding (7) and integration (6).
Only 33% (20 of 60) of bots were useful on all 10 questions. For 60% (36 of 60) of bots, the usefulness of their best and worst answers was 2 or more points apart on our 0 to 3 scale.
Each question was asked once per bot, so we can't say how much one bot varies when asked the same question twice.
Every vendor, every question
Each row is one of the 60 vendors, sorted by how many of the 10 questions got a complete, accurate answer. Vendors are not named.
Data table
| Outcome | Conversations | Share |
|---|---|---|
| Complete and accurate | 416 | 69% |
| Useful but incomplete | 71 | 12% |
| Complete, but too generic | 4 | 1% |
| Nothing usable | 109 | 18% |
Shares are of all 600 conversations and are rounded.
What does it want in return?
The sales push is real, and pricing draws the most of it
These are sales bots, so we expected them to sell. In 67% (401 of 600) of conversations, the bot asked for contact details or offered a meeting.
| Behaviour | Conversations |
|---|---|
| Asked for contact details or offered a meeting | 67% |
| Asked for contact details | 54% |
| … in its first reply | 43% |
| … before giving any answer | 14% |
| Offered a meeting, demo or call | 33% |
| … instead of an answer | 3% |
| Sent a link | 66% |
| Withheld the answer behind an email gate | 2%, all from one vendor |
By topic (of 120 each):
| Topic | Asked for contact | Asked before answering | Offered a meeting | Sent a link |
|---|---|---|---|---|
| Pricing | 79% | 31% | 45% | 53% |
| Compliance | 53% | 13% | 24% | 84% |
| Onboarding and migration | 52% | 4% | 28% | 63% |
| Integration | 49% | 4% | 37% | 79% |
| Competition | 38% | 18% | 32% | 50% |
Bold marks the highest and lowest value in each column.
13 of 60 bots asked for contact details in all 10 conversations, and 5 never asked.
Asking first goes with less complete answers on compliance and pricing. On compliance, 8% (1 of 13) of conversations where the bot asked for contact before answering ended with a complete answer, against 41% (23 of 56) where it never asked. On pricing, it was 31% (11 of 35) against 64% (16 of 25). On competition, the pattern reverses: 89% (17 of 19) against 80% (60 of 75). Integration and onboarding had only three ask-first conversations each. Bots that answered first and asked afterwards had the most complete answers on every topic except onboarding, where all three ask-first conversations were complete. These are correlations from one conversation per question, not proof that asking first causes worse answers.
| Topic | Asked before answering | Asked after answering | Never asked |
|---|---|---|---|
| Compliance | 8% | 67% | 41% |
| Pricing | 31% | 93% | 64% |
| Competition | 89% | 92% | 80% |
| Integration | 2 of 3 | 91% | 87% |
| Onboarding and migration | 3 of 3 | 91% | 48% |
The vendor that gated every conversation is excluded.
Bold marks the highest and lowest value in each column. Cells with fewer than 5 conversations aren't marked.
03
Why sales chatbots fail: Patterns we found
vendors fell back on the same "couldn't find specific information" reply.
conversations from the same vendor ended at an email wall.
bots hedged weakly on an integration that doesn't exist.
of bots gave a complete, accurate answer on billing terms when their pricing page was silent.
Misreading the question
A business communications software vendor answered "how does onboarding work?" by describing its own employee onboarding.
Customer
ok, but what are the actual steps?Chatbot
The onboarding process at [vendor] is designed to be engaging and comprehensive. It typically includes:- Virtual Orientation Sessions: New hires participate in live virtual orientation to get acquainted with the company culture.
- Interactive Modules: These include tutorials, videos, and guides, delivered over the first month to help employees settle in. [...]
Annotation: Describes onboarding for its own new employees, not for a customer.
A canned wrong answer, repeated
A business software vendor answered a SOC 2 question, a data residency question and a CRM integration question with the same line, twice in each conversation. Our buyer replied "that's about incidents, not what i asked." The bot sent the same line again.
Customer
we're evaluating your product. are you soc 2 compliant, what type, and how do we get your current certificate?Chatbot
Thank you for asking. Full information about any current security incident is available on our Status page at [link]Customer
that's about incidents, not what i asked. do you have a soc 2 type ii certificate, and is there a trust page or security page where a prospect can get it?Chatbot
Thank you for asking. Full information about any current security incident is available on our Status page at [link]Annotation: The same off-topic reply, sent again after the buyer pointed out the mistake.
Loops
A healthcare software vendor said yes to an integration question, then asked our buyer three times what they were most interested in, instead of explaining how the integration works.
Customer
ok, but how does it actually work?Chatbot
To help us provide the most relevant and tailored information, could you please let me know what you're most interested in? [...]Customer
not sure yet, our it team handles the details. i just need to know how it connects with [partner system], what's the basic mechanism?Chatbot
To ensure we provide you with the best possible assistance tailored to your needs, could you let us know what you're most interested in? [...]Annotation: The buyer answers the question and asks again. The bot asks the same thing again.
The "couldn't find" fallback
When a bot didn't know, it often said so with a stock line and pointed to sales. "Couldn't find specific information" came from 16 vendors, 12 of them on platform B.
Customer
we're evaluating your product. how does onboarding work, and how long does it usually take?Chatbot
I couldn't find specific details on the onboarding process. [...]Customer
ok, but what are the actual steps?Chatbot
I couldn't find specific details on [vendor]'s onboarding steps. Would you like to discuss something else or explore how our solutions can benefit your business?Annotation: No answer, twice, and no path to one.
Silence on the website
When the website is silent, the bot usually is too. On billing terms, bots whose pricing page stated the terms gave a complete, accurate answer 73% (16 of 22) of the time. Bots whose page said nothing did so 21% (7 of 34) of the time.
The email wall
One finance software vendor answered nothing until the buyer gave an email address, in all 10 conversations.
Chatbot
Great, let's get you connected with our team right now. [...] First, what's your email address?Customer
i'd rather not share that yet, i just need to know if monthly is an optionChatbot
Please enter a valid email addressAnnotation: The bot accepts nothing but an email address. Every one of this vendor's conversations ended here.
Weak hedges
Asked about an HR system that doesn't exist, 57 of 60 bots said they couldn't confirm it and pointed to a real path such as an API or file import. Four bots hedged weakly.
Customer
we're evaluating your product. does it integrate with [fictitious product]?Chatbot
I can help check that. I’m not seeing a published, out-of-the-box [vendor] connector specifically for [fictitious product]. [...]Customer
is there any other way to connect them?Chatbot
Yes, in a lot of cases you can still connect them even without a prebuilt “[fictitious product]” connector, depending on what you’re trying to pass between the systems. [...]Annotation: Honest at first, then a leading "Yes" about a system the bot can't know.
04
What good looks like: Examples worth copying
Answer first, then ask, and say why
The best bots answered the question, then asked for contact with a reason tied to it. 31% (185 of 600) of conversations did this.
Customer
we're evaluating your product. are you soc 2 compliant, what type, and how do we get your current certificate?Chatbot
Yes. [vendor] undergoes independent SOC 2 audits and states it has completed a SOC 2 Type II examination (Type II).- Type: SOC 2 Type II
- How to get the current report/certificate: [vendor] notes the SOC 2 report is available to clients and partners upon request and completion of a non-disclosure agreement (NDA). [...] If you want, I can help with the next step by setting up time with a [vendor] representative so you can request the current SOC 2 report under NDA. Would you like to do that?
Annotation: Answers all three parts, then offers a next step tied to getting the report.
Questions that make the answer better
Only 6% (37 of 600) of conversations had a bot ask about the buyer's situation before answering and then use the reply. When it happened, the answer was specific.
In 51% (304 of 600) of conversations, the bot's context question came after the answer. In 15% (92 conversations), the bot asked before answering but didn't use the reply. A question that refines the answer can only happen in a conversation that got an answer.
Customer
roughly what would that come to for about 200 people?Chatbot
Good question, and “200 people” can change the ballpark a lot depending on what you mean. [...] can you confirm which one you mean: 200 seats, or 200 monthly responses?Customer
200 internal users. most of the company would use it. roughly what would that come to?Chatbot
Got it, 200 internal users usually points to an Enterprise setup, since self-serve plans are generally designed for smaller teams [...]Annotation: One clarifying question, and the answer changes with the reply.
Candid answers about the competition
Asked what the main competitor does better, 83% (50 of 60) of bots gave a complete answer. 30 of them named specific cases where they're not the right fit.
Customer
ok, then where are you not the best fit?Chatbot
Fair question. [vendor] is usually not the best fit when:- You want a simple personal to-do list only and nothing needs to roll up to team workflows, reporting, or goals.
- Your work is mostly real-time chat support and you need a full helpdesk stack (live chat, call centre features). [...]
Annotation: Names concrete cases where a buyer should look elsewhere.
A clean "I can't confirm", with a real path
A software development tools vendor, asked about the system that doesn't exist, said it couldn't see a listed, native integration. It then pointed to single sign-on, user provisioning and its API as the usual ways to connect.
Customer
we're evaluating your product. does it integrate with [fictitious product]?Chatbot
I’m not seeing a listed, native [vendor] integration specifically for [fictitious product]; typically teams connect HRIS systems via SSO/SCIM through an identity provider or via API/webhooks [...]Annotation: Says what it doesn't know, then gives the real ways to connect.
Direct answers on data residency, with the conditions
An enterprise software vendor answered directly whether EU hosting is the default, and added that the region is picked when the account is set up.
Customer
i'd rather not share that yet. is eu hosting the default, or something we have to ask for?Chatbot
It’s something you choose, not something I’d assume is the default. In practice, EU hosting is selected during account setup/provisioning so your tenant is created in our EU region (we support US and EU regions). [...]Annotation: A direct answer to "default or on request", with when the choice is made.
05
The sales side: What happens after the answer
When the bot hands you to sales
33% (199 of 600) of conversations included an offer of a meeting, demo or call. In 3% (20 of 600), the offer came instead of an answer. Platform B bots offered a meeting instead of an answer in 16 of 160 conversations, against 4 of 440 for platform A.
When the buyer says no
Our buyer was scripted to decline to share contact details. 73 conversations asked for contact before answering, not counting the vendor that gated everything (see "The email wall" in section 03). 89% (65 of those 73) still went on to answer, partly answer or point to sales. A data software vendor's bot asked for an email first, then, after the buyer declined, replied: "Understood, and you don’t need to share an email to get a general answer."
The same sentences, from unrelated vendors
Some replies were word-for-word identical across vendors that have nothing to do with each other:
- The sign-off "If you have any more questions in the future, feel free to reach out. Have a great day!" came from 13 vendors, 9 of them on platform B.
- "You’re welcome, glad that helped" closed conversations for 13 vendors, all on platform A.
- On the fictional system, five vendors gave the same line: they "couldn't find specific information about an integration between" themselves "and [fictitious product]".
OUR INFERENCE
These vendors may be using default wording provided by their chat platforms, without customizing it.
06
Why these answers matter to your pipeline
OUR INFERENCE. Not measured in this study.
The highest-intent question gets the weakest answer.
Monthly vs annual billing got a complete, accurate answer in 43% (26 of 60) of conversations, the lowest of the 10 questions. Pricing is the question closest to a buying decision. A weak answer here has worse consequences than anywhere else.
Your chatbot can disqualify you.
A buyer who gets a clear answer can shortlist you without a call. A buyer who gets "I couldn't find specific information" may not come back, and may do their research elsewhere instead, including in public AI chatbots.
The best sales bots are starting to act like an SDR rather than a Q&A engine.
They ask about the buyer’s situation, tailor the answer and suggest a next step that doesn’t feel like a gate. But it’s still rare: only 3% (17 of 600) of conversations did all three.
07
Five actions to improve your sales chatbot
- 1
Put billing terms and other common questions on your website, and check that your bot uses them.
Bots gave a complete, accurate answer on billing terms 73% of the time when the pricing page stated them, against 21% when it didn’t. But 27% (6 of 22) still missed an answer that was on the page.
- 2
Rewrite the default template phrases.
Identical sentences turned up across dozens of unrelated vendors. Make your bot sound like your brand.
- 3
Answer first, then ask, and give a reason.
Answer the buyer's question before asking for contact details or a meeting, and say why you're asking. On compliance, bots that asked for contact before answering gave a complete answer 8% (1 of 13) of the time, against 41% (23 of 56) for bots that never asked. On pricing, it was 31% (11 of 35) against 64% (16 of 25). These are correlations, not proof.
- 4
Test every topic, not just the demo question.
21% of paired questions split between a useful answer and an unhelpful one from the same bot.
- 5
Move from answering questions to running a sales conversation.
The strongest bots asked about the buyer’s situation, tailored the answer and offered a next step with a clear reason. Only 3% (17 of 600) of conversations did all of that. Treat it as a major investment, with its own roadmap.
A step-by-step guide to test your own chatbot is in the full report.
08
Research methodology
Vendors and chatbots tested
60 B2B software vendors, from under 1,000 employees to over 10,000, with an AI sales chatbot on their marketing site. We removed one bot that offered only buttons, with no way to type a question. Six conversations where a bot replied only with buttons are kept and counted as nothing usable. The bots ran on two different chat platforms, which we call platform A (44 bots) and platform B (16 bots).
| Vendor size | Vendors | Share of vendors |
|---|---|---|
| Under 1,000 employees | 10 | 17% |
| 1,000 to 2,999 employees | 31 | 52% |
| 3,000 to 9,999 employees | 15 | 25% |
| 10,000 or more employees | 4 | 7% |
| Total | 60 | 100% |
Shares are rounded, so they may not add up to exactly 100%.
Five topics, 10 questions
| Topic | Objective | Variation A | Variation B |
|---|---|---|---|
| Compliance | Does the vendor have the security and data assurances a buyer needs, and how does a prospect get the details? | Asks whether the vendor is SOC 2 compliant, which type, and how to get the current "certificate" | Asks where the data is stored, because it has to stay in the EU |
| Pricing | Can a buyer build a rough budget before talking to sales? | Asks what the pricing is based on | Asks whether there's a monthly option, or only annual |
| Competition | Can a buyer get a straight comparison with the vendor's main competitor? | Asks how the product differs from the main competitor | Asks what the main competitor does better |
| Integration | Can a buyer trust the bot's answers about connecting to their systems? | Asks about a system the vendor lists as supported, and how the integration works | Asks about a system that doesn't exist (a fictitious product we invented for this study) |
| Onboarding and migration | Can a buyer learn what adoption will take before talking to sales? | Asks how onboarding works and how long it usually takes | Asks how to migrate existing data into the product |
The competition questions named one of each vendor's main competitors, and the integration question named a system the vendor lists as supported on its own website. Neither name appears on this page.
Each bot got each question once: 600 conversations, run between October 30 and November 6, 2026. The simulated buyer played a prospect at a company of about 200 people, opened with the question as written above, and followed up the way a buyer would.
How we ran the conversations
Isalo is an AI-native managed service that continuously tests, monitors and optimizes the digital experiences customers and buyers rely on. For this research, we used isalo's own agents to run and grade every test.
An isalo agent played the buyer in every conversation: an anonymous operations or IT lead at a company of about 200 people. It asked the opening question, read each reply and decided what to say next, the way a real buyer would: answering qualifying questions, following up on a vague answer, and declining to share contact details or book a meeting. It stopped before a live person joined.
Each bot got each question once, so there was one conversation per question per bot. Some chatbots were tested through headless automation and others through browser-assisted interaction. The same agent made every conversational decision either way, so the buyer's side of the conversation was the same whichever method we used.
How we graded
Every conversation was graded against a written rubric for its question. Two tests then placed it in one of four outcomes: did the bot answer completely, and was the answer useful and accurate?
| Outcome category | Definition | Evaluation criteria |
|---|---|---|
| Complete and accurate | The bot gave every item the rubric requires, the answer was specific enough for a buyer to act on, and nothing it said is contradicted by the vendor's public pages. | No contact details or meeting needed to get the answer |
| Useful but incomplete | The answer was specific and accurate, but missed something the rubric requires. | For example SOC 2 confirmed without the report type, an honest "can't confirm" with a real path, or a good answer delivered as one unbroken block of more than about 90 words |
| Complete, but too generic | The bot covered every item the rubric requires, and nothing it said is contradicted by the vendor's public pages, but the answer was too generic to act on. | Generic: a competitor in the same category could say the same sentence about itself |
| Nothing usable | The bot gave the buyer nothing they could rely on: no specific answer, or an answer that includes a claim the vendor's public pages contradict. | Includes vague or generic partial answers, routing to sales or offering a meeting instead of answering, an email gate, and the same reply repeated without answering |
Each question had its own rubric. For example, a SOC 2 answer counts as complete only if it says whether the vendor has a SOC 2 report, which type, and how a prospect can get it.
Two more checks ran on every conversation. Usefulness: how much a buyer could act on the bot's best reply, on a scale from 0 to 3. Accuracy: every specific claim was checked against the vendor's public pages on November 6, 2026. An answer counts as useful when it scores at least 2. "Complete and accurate", the headline measure, needs all three: complete, useful, and not contradicted.
To check the grading, a second, independent AI grader re-graded 12% (72 of 600) of the conversations blind, across all five topics, without seeing the final grades. It matched our grade on 81% of conversations, and the outcome group on 88%.
Research limitations
- Testing was anonymous and unauthenticated. The simulated buyer never signed in and never shared contact details.
- The five topics were selected for relevance across vendors, rather than customized to each vendor. Three questions used a vendor-specific name: the main competitor in both competition questions, and a system the vendor lists as supported in the first integration question.
- Not every answer could be independently verified. Specific claims were checked against each vendor's public pages only, and 47% (205 of 438) of checkable claims couldn't be confirmed or contradicted. A claim we couldn't confirm may still be true.
- For selected topics, including SOC 2, EU data residency, pricing, integrations, onboarding and data migration, additional checks were performed against vendor trust centers, pricing pages and product documentation. Claims bots made about competitors were not verified.
- Findings represent the observed test period and should not be interpreted as permanent vendor rankings.
- The buyer was a simulated AI agent, not a person.
- We tested the sales chatbot on each vendor's marketing site. Chatbots on support pages or inside the product weren't tested.
- Each bot was asked each question once, so we can't measure how much a bot varies on the same question.
- We stopped at the handoff to a person, and didn't measure anything that happened after it.
- The bots ran on two different chat platforms. We don't name them, and we make no claim about why their results differ.
- Results describe the test window, October 30 to November 6, 2026.