How to Audit Your Chatbot After 30 Days (Simple Checklist)
You set up a chatbot a month ago. It answered a few questions, maybe deflected some support tickets, and now it just... sits there. Sound familiar?
The first 30 days are when a chatbot either proves itself or quietly becomes dead weight on your site. The problem is most small business owners never actually sit down and check. They assume it's working because it's turned on.
This guide walks you through how to audit your chatbot after 30 days — in plain English, no dashboards-and-webhooks talk. We'll cover what numbers to pull, what questions to ask, and how to decide whether to keep, fix, or replace your bot. Tidio, Crisp, and Freshchat each expose these numbers differently, so we'll point out where each one makes the audit easier or harder.
Prices and limits below come from each vendor's public pricing page and product documentation. This is a desk review, not a paid hands-on test.
Start with the one number that matters: resolution rate
Resolution rate is the share of conversations your bot handled start to finish without a human stepping in. If 100 people chatted with your bot last month and 40 got their answer without help, that's a 40% resolution rate.
Pull this number first. Everything else is secondary.
No vendor publishes a benchmark for this, so treat any figure you see quoted as a rule of thumb rather than a measured average. The more useful question is *why* the other conversations needed a human. Sort your bot's "couldn't help" conversations into topics and look at the biggest bucket. It's often one subject the bot was never trained on — custom pricing, a service you don't list on the site, a policy buried in a PDF. That's not a bot problem. That's a content gap you can usually fix in an afternoon.
Where to find it:
- **Tidio:** Check the Lyro AI analytics tab for AI-handled vs. human-handled conversations. Note that deeper analytics are locked to higher tiers, so on cheaper plans you may only see the basics.
- **Crisp:** The shared inbox lets you filter conversations by bot vs. human. It's less automated than Tidio's reporting, so expect to count manually.
- **Freshchat:** Freddy AI reporting is solid but the best views sit on higher plans. On the free tier (up to 10 agents), expect a stripped-down picture.
Read 20 real conversations — yes, actually read them
Numbers tell you *what* happened. Reading transcripts tells you *why*. Set aside 30 minutes and read the last 20 bot conversations end to end. You're looking for three things:
- **Misunderstandings.** Did the bot answer a question nobody asked? That's a training problem.
- **Dead ends.** Did the bot say "I don't know" and leave the customer hanging? That's a handoff problem.
- **Tone misses.** Did the bot sound robotic when the customer was frustrated? That's a personality problem.
The pattern to look for: most "bad" bot conversations aren't technical failures at all. They're the bot confidently answering the wrong question. A customer asks "do you ship to Canada?" and the bot replies with the return policy because it matched on the word "ship." Fixing that is usually a matter of adding a few example phrases, not rebuilding anything.
If you're on Freshchat, filter transcripts by "resolved by bot" so you're not wading through human chats. On Crisp, use the tag system to mark conversations as you read them — it makes the next audit much faster.
Check what the bot actually cost you
Chatbot pricing is sneaky. The sticker price is rarely the real price. Month one is the right time to reconcile what you *thought* you'd pay against what you were actually billed.
Here's how the three compare on published pricing:
- **Tidio:** Plans start at $24.17/mo billed yearly, with a free plan covering 50 conversations/mo. Lyro AI gives you the first 50 AI conversations free (lifetime), but credits run out quickly on cheaper plans. If your bot is chatty, this is where costs creep.
- **Crisp:** Flat €45/workspace/mo pricing, which works out cheap for big teams because it isn't per seat. The catch: the free plan has zero AI credits, and AI features are lighter than Tidio's. If AI is central to your bot, budget for it.
- **Freshchat:** From $19/agent/mo billed yearly, with a free plan up to 10 agents. Freddy AI Agent is billed per session as a paid add-on — the first 500 sessions are free, then $49 per 100 sessions. That's the number to watch if your volume is high.
Do the math in your own currency and your own numbers: take your actual conversation count from month one, project it to your busiest realistic month, and check where each vendor's caps and overage charges land. If the projection scares you, that's useful information *now*, not in month six.
Test the handoff to a human
This is the step almost everyone skips, and it's the one customers notice most. When your bot can't help, what happens next?
Run this test yourself: open your site in a private browser, ask the bot something it definitely can't answer, and time how long it takes to reach a real person. Then check whether that person could see the conversation history.
- **Tidio** handles this well out of the box — the multi-channel inbox keeps bot and human chats in one thread.
- **Crisp** does it through its shared inbox, which is the core of the product. Solid, if unglamorous.
- **Freshchat** shines here if you're already in the Freshworks ecosystem — the bot hands off into the full ticketing suite.
If your handoff is clunky, fix it before you do anything else. A great bot with a broken handoff feels worse to customers than no bot at all.
Look at what customers do *after* the chat
A bot that answers questions but doesn't move anyone toward a purchase, booking, or signup is a cost center. Pull your conversion data for the last 30 days and compare chat users vs. non-chat users.
You're asking: did people who chatted with the bot buy more, buy less, or the same? Did they come back? Did they open a support ticket anyway?
Don't overthink the analytics. Even a rough comparison — say, average order value for chat vs. non-chat visitors — tells you whether the bot is helping or just hovering. If chat users convert at the same rate as everyone else, your bot is decoration.
Decide: keep, fix, or replace
After 30 days you should have enough to make a call. Here's the simple framework to use:
- **Keep** if resolution rate is climbing, transcripts look sane, handoffs work, and costs are predictable.
- **Fix** if the bot is close but failing on specific topics — that's a training problem, not a tool problem. Add the missing content and re-audit in 30 days.
- **Replace** if the tool itself is the bottleneck. If you're on Crisp and AI is central to your strategy, its lighter AI features may not cut it — Tidio or Freshchat may fit better. If you're on Tidio and analytics matter more than price, the higher tiers unlock what you need. If you're on Freshchat and per-session AI billing is eating you alive, a flat-rate tool like Crisp becomes attractive.
Be honest with yourself here. Most "replace" decisions are really "fix" decisions in disguise. Give the bot one more month before you switch.
Frequently asked questions
How long should I wait before auditing my chatbot?
Thirty days is the sweet spot. It's long enough to gather real conversation data and short enough that problems haven't calcified. If your traffic is very low — say, under 50 conversations a month — wait 60 days so you have enough to actually analyze.
What's a good resolution rate for a small business chatbot?
No vendor publishes an official benchmark, so treat any quoted range as a rule of thumb rather than a measured average. What matters more is direction: a rate that climbs month over month means your content fixes are landing. A rate that stays flat in the low double digits usually means the bot was never trained on the questions customers actually ask — fix the content before blaming the tool.
Do I need a technical person to run this audit?
No. Everything in this guide can be done from the tool's dashboard. Tidio and Crisp are especially beginner-friendly. Freshchat is a bit denser, but if you can read a spreadsheet and click through a chat history, you can run this audit yourself.
Should I switch tools if my first audit looks bad?
Usually not yet. One bad month is normal — most bots improve dramatically after their first round of fixes. Switch only if the tool itself is holding you back, like AI features being too limited on your plan or per-session billing making costs unpredictable. Otherwise, fix the content and re-audit in 30 days.
Ready to pick one?
Marcus Reed — Marcus Reed is the senior editor at ChatbotRadar. He tracks live chat and AI chatbot tools for small teams, verifying pricing on vendor sites and trialling tools on their free plans. How he tests · Report an error
Disclosure: this article contains affiliate links. If you buy through them we may earn a commission at no extra cost to you. Pricing and features are re-checked regularly but can change — always confirm on the vendor's site.