Finding Chatbot Intent Gaps in Fallback Transcripts

Collect the events that show the bot missed

Export fallback events, repeated rephrasing, explicit requests for a person, and conversations abandoned after a bot reply. Include the message text, the detected intent, the confidence, and the next action. A containment dashboard alone will not show why the bot failed.

Deduplicate near-identical messages before labeling, but keep the frequency count. Ten variations of where is my order are one intent gap with ten occurrences, not ten separate findings. The count is what makes prioritization possible.

Label the customer goal, not the bot's guess

Read the message and write the goal in plain language: change delivery address, understand a charge, cancel a subscription, recover access. Compare it with the intent the bot selected. The gap may be a missing intent, a mislabeled intent, or an answer that exists but is hard to reach.

Use a small label set and a separate note for ambiguous cases. If two reviewers disagree, keep the example and discuss it rather than forcing a label. The disagreement often reveals a phrase the bot should recognize or a question the content does not answer.

Use the specialist service role as a starting point for scope, sources, and evidence.

Separate content gaps from flow gaps

A content gap means the answer does not exist or is wrong. A flow gap means the answer exists but the bot does not offer it at the right moment. The repairs differ: content needs an owner and a source; flow needs a dialogue change, a button, or a shorter path.

Check whether the human queue solved the issue with an existing macro. If so, the knowledge exists and the bot failed to surface it. If the agent had to escalate, the gap may be a policy or system limitation that no chatbot rewrite can fix.

Prioritize by volume, risk, and fix cost

Rank gaps by how often they occur, the harm if mishandled, and the effort to repair. A rare question about a legal notice may outrank a frequent question about store hours. A high-volume gap with an easy content fix is usually the first release.

Do not prioritize by the loudest stakeholder or the most elegant technical solution. The review should show the counts and the proposed repair so the decision is visible. If a gap requires a new integration, label it as a project rather than a content ticket.

GOV.UK Service Manual: helping people to use your service provides public guidance that can help your team test workflow assumptions.

Retest with the same phrases and a fresh sample

After the fix, replay the original messages and check whether the bot now routes correctly. Then test a fresh sample from the same period to see whether the change generalizes. A bot that only recognizes the exact phrases from the review has been overfitted.

Measure resolution and handoff quality, not just containment. A bot that keeps a frustrated customer in a loop without a human has contained the chat and failed the person. Pair containment with repeat contact and satisfaction data.

Compare this approach with the next related workflow guide.

Keep a living gap register

Record the gap, examples, count, owner, repair, release date, and retest result. Review it monthly with support, content, and product. Retire entries that are fixed and add new ones from the latest transcripts.

Share a short summary with the support team so agents know what changed and what still needs a human. The register is also useful evidence when someone proposes expanding the bot's scope: it shows which intents are ready and which are still fragile.

Common questions

Ready to create a clearer support role?

Bring the workflow that consumes the most attention. We will help shape the role, handoff, and review points.