June 27, 2026

AI chatbots for customer service: what actually works after 90 days in production

Ninety days in, the polished demo bot looks nothing like what's running in production. Here's what we kept, what we ripped out, and what nobody warned us about.

KEY TAKEAWAYS

↳Containment rate is a vanity metric. Track resolved-without-human and CSAT on bot-only tickets instead.
↳RAG quality collapses when your help docs are stale. Fix the docs before you fix the model.
↳The handoff to a human is the product. If that flow is bad, the bot is bad, full stop.
↳Small businesses don't need GPT-5. They need a tight scope, three tools, and a working escalation path.
↳Free tier chatbots are fine for FAQs. They are not fine for anything that touches an order, a refund, or a calendar.

We put our first production support bot in front of real customers on a Tuesday in February. By Friday it had cheerfully told someone their refund was processed when it absolutely was not. By the following Monday we’d rewritten the system prompt three times, added two guardrails, and started logging every single tool call to a Postgres table I now check more often than my email.

Ninety days in, I have opinions. Some of them I did not expect to have.

This is a post about what actually survives contact with real users when you ship AI agents for support and lead-gen. Not the demo. The thing that runs on a Tuesday at 2am when someone in Perth is angry about a double charge.

The demo-to-production gap is bigger than you think

Every customer service AI chatbot demos beautifully. You ask it three reasonable questions, it answers three reasonable questions, everyone claps. Then you put it in front of 4,000 real tickets a month and discover that roughly 18% of your users do not type in complete sentences, 6% are testing it for jailbreaks because they are bored, and a meaningful chunk are pasting in screenshots that your bot cannot see.

The gap between “works in the playground” and “works on Tuesday at 2am” is mostly:

Messy inputs (typos, half-sentences, mixed languages, screenshots)
Stale or contradictory help docs
Edge cases your founder forgot to mention (the legacy plan from 2022 that still has 80 paying customers)
Tool calls that fail silently because Stripe rate-limited you

None of that shows up in a demo. All of it shows up in week two.

Containment rate is mostly a lie

Every vendor sells you on containment rate. “Our bot handles 70% of tickets.” Sure. So does a wall.

The number I actually care about is resolved without human intervention AND positive CSAT on the bot-only conversation. When we started tracking that instead, our “70% containment” bot was actually resolving maybe 41% of tickets in a way the customer felt good about. The other 29% were closing the chat in frustration, not getting helped.

We now log three things per conversation:

type BotOutcome = {
  resolved: boolean;          // did we close the ticket without a human?
  csat: 1 | 2 | 3 | 4 | 5 | null; // post-chat survey
  escalated_to: 'human' | null;
  tool_calls: { name: string; ok: boolean; ms: number }[];
  cost_usd: number;
};

That last field matters more than people admit. One of our clients was running a Claude 3.5 Sonnet bot that cost them $0.34 per resolved ticket. After a rewrite that pushed FAQ traffic to a cheaper model and reserved Sonnet for tool-using flows, it dropped to $0.09. Same CSAT. That’s the work.

Your RAG is only as good as your help docs

This is the unsexy lesson. Half the support bot projects I’ve seen fail are not model problems. They are documentation problems.

If your help center has three articles about refunds that contradict each other (one from 2021, one from 2023, one written by a contractor who left), your bot will confidently pick whichever chunk has the highest cosine similarity to the user’s question. That’s usually the wrong one.

Before we ship a support bot for a client now, we do an audit. Last month we went through 312 help articles for an e-commerce client and found 47 that were either stale, contradictory, or referenced features that no longer existed. We deleted 31 and rewrote 16. The bot’s accuracy on refund-related questions went from 68% to 91% without changing a single line of prompt.

Fix the docs. Then fix the bot.

The handoff is the product

I cannot stress this enough. When the bot can’t help, what happens next is the entire product.

Bad handoff: “I’m sorry, I can’t help with that. Please email support@.” User now has to re-explain everything. Rage.

Good handoff: bot summarizes the conversation, tags the ticket with the inferred category, attaches the user’s order ID it already looked up, and routes to the right human queue. The agent who picks it up has context in three seconds.

We build this with a small tool the bot calls when it decides to escalate. It writes to the helpdesk (Front, Help Scout, whatever) with a structured payload. The bot’s last message to the user is something like “I’ve passed this to Sarah on our team with everything we talked about. She’ll reply within two hours.” That sentence alone moved CSAT on escalated tickets up by almost a full point for one client.

What an ai chatbot for small business actually needs

If you run a 4-person business and you’re shopping for an ai chatbot for customer service, free tier or otherwise, here’s the honest version.

You do not need GPT-5. You do not need a multi-agent system. You need:

A bot that knows your top 20 FAQs cold
The ability to look up an order, a booking, or an account by email
A clean escalation to a human (even if that human is you, on WhatsApp)
Logs you can actually read

The free tiers (Intercom Fin’s trial, Tidio’s free plan, the open source projects on GitHub like Chatwoot or Botpress) will get you to step 1. Steps 2 through 4 are where you either build something custom or pay for a tier that supports tools and webhooks. For most small businesses, the math works out to roughly $80-300/month for something that genuinely deflects support load, vs. $0 for something that just answers FAQs and pisses people off when they have a real problem.

The “ai chatbot for customer service github” search results are mostly Chatwoot, Botpress, Typebot, and a long tail of weekend projects. Chatwoot plus a custom LLM integration is a legitimate path if you have one engineer who likes Ruby. Otherwise the self-hosting tax (Hetzner box, Postgres, vector DB, monitoring, the 2am pages) eats whatever you saved on SaaS fees.

What we ripped out

Things that sounded smart in week one and were gone by week six:

Long, elaborate system prompts. We had a 2,400 token system prompt. We’re now at 680. The bot is better. Models follow short, specific instructions more reliably than long, hedged ones.
Letting the bot freestyle apologies. Every “I’m so sorry to hear that” was costing tokens and sounding insincere. We templated the empathy and let the model focus on actually solving the thing.
“Personality.” Nobody wants their refund bot to be quirky. Users want competence. We dialed the personality knob to zero and complaints went down.
Confidence scores shown to users. Tried it for a week. Confused everyone. Killed it.

What we kept

Aggressive logging. Every tool call, every retrieval, every model response, stored and queryable.
A weekly review where someone (a human) reads 50 random conversations. This is the single highest-leverage thing we do. It catches drift, bad docs, and edge cases nothing else surfaces.
A kill switch. One env var flips the bot off and routes everything to humans. We’ve used it twice. Both times we were glad it existed.
Turnstile on the chat widget. Bots talking to bots is a real and boring problem.

The honest summary

AI chatbots for customer service work. They also fail in ways that are specific, predictable, and mostly fixable if you actually watch what they do. The teams getting value are not the ones with the fanciest model. They’re the ones who treat the bot like a junior employee: clear scope, good docs, a manager who reviews their work weekly, and a clear path to escalate when they’re stuck.

If you’re thinking about putting one in front of your customers and you want a second opinion on the scope before you spend three months building the wrong thing, come say hi. I’ll tell you if I think it’s a good idea.

FREQUENTLY ASKED

Common questions

▸How long does it take to get a customer service AI chatbot to actually work in production?

Plan for 60 to 90 days from kickoff to something you trust unsupervised. The first two weeks are setup and a closed beta. Weeks three through six are where you find the real edge cases and rewrite half your prompts and retrieval logic. By day 90 you should have stable metrics, a working escalation flow, and a weekly review cadence. Anyone promising a week-long deployment is selling you the demo, not the production system.

▸Is there a genuinely good free AI chatbot for customer service?

For pure FAQ deflection, yes. Tidio's free tier, Chatwoot self-hosted with an OpenAI key, and Botpress all work. The catch is they hit a wall the moment you need to look up an order, check a booking, or do anything account-specific. If your support load is mostly "what are your hours" type questions, free is fine. If customers ask about their stuff, budget for a paid tier or a custom build.

▸What metrics should I track besides containment rate?

Track resolved-without-human combined with positive CSAT on bot-only conversations. Track cost per resolved ticket. Track escalation quality (did the human get useful context?). Track time-to-first-helpful-response. Containment rate alone rewards bots that frustrate users into giving up, which is the opposite of what you want.

▸Should I build my own or use a vendor like Intercom Fin or Zendesk AI?

If your support volume is under maybe 1,000 tickets a month and your needs are standard, use a vendor. The integration work is done, the UI is solid, you get analytics. Build custom when you have unusual tool integrations, strict data residency needs, a workflow no vendor supports, or volume high enough that per-resolution pricing gets painful. We build custom for clients where the vendor math stops working, usually around 5,000 plus tickets a month or weird compliance constraints.

▸How do I stop the bot from hallucinating policies or making things up?

Three things, in order. First, ground every factual claim in retrieved documents and refuse to answer when retrieval confidence is low. Second, keep tool outputs (order lookups, account data) as the source of truth and instruct the model to quote them verbatim, not paraphrase. Third, log every response and review 50 a week. Hallucinations are usually a documentation problem or a retrieval problem, not a model problem.

▸What's the smallest useful AI chatbot for a small business?

A bot that knows your top 20 FAQs, can look up one thing (an order, a booking, an account), and hands off cleanly to a human when it's stuck. That's it. You can build this in two to three weeks with a vendor platform, or in a week with a focused custom build if you already have clean help docs. Everything beyond that is optimization, not core value.

5/28/2026