Containment is not resolution — how to measure what your AI agent actually did

An operator opened a monthly report that said 82% of conversations were "handled without a human." The same week, a 1-star review appeared: the bot had told a customer that next-day installation was available in a zip code the company does not serve. Both facts were true at once. The dashboard number answered the question did a human join the chat. The review answered the question did the customer get what they came for. Only the second one predicts whether the tool keeps its contract.

Why containment flatters itself

Containment means a conversation ended without escalating to a human. That is all it means. It measures the agent's willingness to keep talking, not whether anything correct happened.

Three conversations can all be contained, and they are not the same outcome: one was genuinely resolved; one ended because the customer gave up after two useless replies; one ended with the customer confidently misinformed. The dashboard counts all three as wins, because from the system's point of view nothing required a human — and that is exactly the problem. The third conversation cost more than a human would have.

Containment is also easy to inflate without anyone technically lying: make the "talk to a person" option harder to find, let the bot re-ask its clarifying question one extra time, nudge customers toward a form instead of an agent. Every one of those raises the containment rate and lowers the quality of the containment. A number that rises when you make the product worse is not a metric you can manage by.

Contained versus resolved — the definition that matters

Resolution cannot be read off a counter. It needs judgment applied to real transcripts, which means it needs a sample. That is the whole method: a small, honest, repeatable sample beats a large, automatic, flattering number.

How to sample transcripts and measure it

Once a week, pull 30 transcripts at random — or all of them if you had fewer. Exclude obvious spam and test messages. Read each one and tag exactly one outcome:

• Resolved — the question was answered correctly, and the transcript shows the customer got it (a confirmation, a booking, a coherent next message).
• Contained, unresolved — no human joined, but the question was not actually answered or was answered wrongly.
• Escalated — handed to a human. This is not a failure; it is the escalation path doing its job.
• Abandoned — the customer stopped responding mid-thread.

Write the tags down every week, in the same file, with the same definitions. A week looks like this:

And the scoring block:

WEEK [x] RESOLUTION SAMPLE — n = 30 random transcripts
Resolved ........................ [ ] /30
Contained, unresolved ........... [ ] /30
Escalated ....................... [ ] /30
Abandoned ....................... [ ] /30

Resolution rate = Resolved / 30 = [ ]%
Containment rate = (Resolved + Contained-unresolved + Abandoned) / 30 = [ ]%

Two rules keep the numbers honest. First, resolution rate is resolved divided by all sampled — no excluding the messy ones. Second, definitions stay fixed. If you redefine "resolved" mid-quarter, every week-over-week comparison you have built becomes decoration.

Three leading indicators that predict trouble before the reviews do

You will not sample every transcript, so you need cheap signals that tell you when to look. These three are visible in any transcript file and they degrade before customers reach a review site.

1. Abandonment after the agent's answer. The customer reads a reply and leaves without acknowledging it. Silent exits after answers usually mean the answer did not answer. Track the count weekly; a rise is the earliest warning you get.
2. The same question phrased two or three ways in one conversation. When a customer re-asks, they are telling you the first answer did not land. Count conversations containing a near-duplicate question; these transcripts are your highest-value reading, because each one points at a specific facts-file gap.
3. "Speak to a human" appearing earlier in conversations. One occurrence per fifty conversations is a working escalation path; the phrase showing up in the first two turns, and climbing week over week, means something in scope or tone recently broke. Find what changed that week.

All three come from text you already have. None of them requires a dashboard you do not.

Reporting the pair honestly to an owner

The report line has both numbers and the sample size, in one sentence:

MONTHLY REPORT — [month]
Contained:      78% of ~900 conversations
Resolution:     24 of 30 random transcripts judged resolved (80%)
Unresolved:     6 — listed below with the fix made for each

The unresolved transcripts, with one line each on the fix, are the most valuable part of the report. They convert the metric into next week's facts-file edits. What you never do is report containment alone: the moment an owner sees one number, the organization optimizes that number, and containment is the one that can be gamed by making help harder to reach.

Four ways teams fool themselves

1. Counting abandoned conversations as contained. The customer left. Nothing was handled. This one accounting choice can swing the headline number by double digits.
2. Sampling only the good transcripts. Pulling the conversations the platform flags as successful is not a sample; it is an advertisement. Random means random.
3. Self-grading without written definitions. If "resolved" means whatever the reader feels that day, the trend line is fiction. Write the four tag definitions down and keep them fixed.
4. Reporting monthly averages that bury one bad week. A terrible week after a pricing change averages into invisibility by month's end. Report the four weekly numbers, not just their sum.

What to do today

• Pull 30 random transcripts from last week and tag them with the four outcomes above.
• Write your four tag definitions in one paragraph at the top of the tracking file, so next week's you grades the same way.
• Set a recurring weekly 30-minute block for the sample — it does not survive as an intention.
• Report both numbers to the owner this month, with the unresolved transcripts and their fixes attached.

Containment tells you what your agent avoided. Resolution tells you what your customers got. Only one of those numbers should ever be allowed near a decision, and it is the one that requires reading.