How to Tell if an AI Agent Is Working: Metrics That Matter

Once an AI agent is answering your phones or chats, a new question takes over: is it actually helping? A busy dashboard full of call counts can look impressive while customers quietly hang up, repeat themselves or call a competitor.

Measuring an agent well does not require a data team. It requires a clear definition of the agent's job, a baseline to compare against, and a routine for looking at outcomes, transcripts, handoffs and customer feedback together.

This guide lays out what to measure, how to review conversations efficiently, and when the numbers should prompt you to pull back.

Start with the job and a baseline

Write one or two sentences describing what success looks like. For example, imagine a small auto-repair shop: "Answer every call, book service appointments for routine work, and get urgent or complicated calls to a service advisor within two minutes."

Then capture a baseline before launch, or pull it from your phone and scheduling records:

• How many calls go unanswered, and when?
• How many calls become booked appointments?
• How quickly do callers who leave messages hear back?
• What do customers complain about today?

Without a baseline, you cannot tell whether the agent improved anything or simply moved the problems around.

Measure outcomes, not just activity

Tag every conversation with an outcome. A simple, consistent set of categories works better than a long list:

1. Resolved by the agent: question answered or task completed correctly.
2. Booked: an appointment or job was scheduled.
3. Handed off: transferred to a person or routed as a message.
4. Abandoned: the customer hung up or left before finishing.
5. Wrong outcome: the agent completed something incorrectly, such as booking the wrong service or giving a wrong answer.

Track these as shares of total conversations, by week, and by time of day. Treat "resolved by the agent" with care. A high rate can hide customers who gave up. The Consumer Financial Protection Bureau (CFPB), in its June 2023 report on chatbots in consumer finance, described "doom loops" in which automated responses fail to resolve a customer's issue and instead lead them "in continuous loops of repetitive, unhelpful jargon or legalese without an offramp to a human customer service representative." A conversation that ends without escalation is not the same as a satisfied customer.

Read the transcripts

Numbers tell you where to look; transcripts tell you what happened. Set a regular review, such as a fixed number of conversations each week, and include a mix: random samples, every "wrong outcome," a share of abandoned calls, and any conversation that triggered a complaint.

Score each one against a short rubric:

• Accuracy: did the agent's answers match your approved information?
• Policy: did it follow your rules on pricing, booking and what it may not discuss?
• Disclosure: did it identify itself as an automated assistant?
• Escalation: did it offer a person when it should have?
• Data capture: were the name, number and details recorded correctly?
• Tone: was it clear, polite and appropriately brief?

Pay special attention to answers that sound right but are not. NIST's Generative AI Profile calls this confabulation, "the production of confidently stated but erroneous or false content," and warns about automation bias, "excessive deference to automated systems." Reviewers should compare the agent's statements against your actual prices, hours and policies rather than trusting that a fluent answer is a correct one.

Watch the handoffs closely

Handoffs are where customers form their opinion of the whole experience. Track:

• Handoff rate, and the reasons behind it. A rising rate for one topic usually means the agent needs better information or that topic belongs with people.
• Time to a person: how long customers wait after asking for help.
• Context transfer: did the person who picked up know what the customer already said?
• Callback completion: for messages taken after hours, how many were returned, and how quickly?
• Missed escalations: conversations where the agent should have handed off and did not. These deserve the closest attention.

The guide "Human in the Loop: Keeping People in Charge of AI Agents" covers how to set escalation rules in the first place.

Ask customers directly

Customer experience is the outcome that ties everything together. Useful signals include:

• A short post-conversation question, such as whether the issue was resolved.
• Repeat contacts: customers who call or message again about the same issue within a few days.
• Complaints and reviews that mention the phone or chat experience.
• Staff feedback from the people who receive handoffs.

NIST's AI Risk Management Framework includes this kind of loop in its MEASURE function, calling for "feedback processes for end users and impacted communities to report problems and appeal system outcomes" to be "established and integrated into AI system evaluation metrics." In plain terms: make it easy for customers to tell you something went wrong, and count those reports alongside your other metrics.

Check what happens downstream

An agent's work shows up later in your systems. Check periodically:

• Do booked appointments show up, or are no-show and cancellation rates higher for agent-booked jobs?
• Are customer records complete and accurate, or full of duplicates and typos?
• Are there double bookings or appointments in slots that should have been blocked?

Make review a routine, and know when to pull back

Monitoring is ongoing, not a launch task. The AI RMF describes post-deployment monitoring plans that include "mechanisms for capturing and evaluating input from users," "appeal and override," "incident response," and "change management." For a small business, that can be a monthly review with a short log of what you changed and why. Re-test after every change to the agent's instructions, information or connected systems.

Decide in advance what would make you pause the agent, such as a cluster of wrong bookings, a missed urgent call or a jump in complaints. The framework calls for mechanisms "to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use."

Finally, transcripts and recordings contain personal information. The FTC's Protecting Personal Information: A Guide for Business advises businesses to "Keep only what you need for your business" and to "Properly dispose of what you no longer need." Set a retention period and limit who can access conversation logs.

Key takeaways

• Define the agent's job in a sentence or two and capture a baseline before launch.
• Tag every conversation with a clear outcome, and be skeptical of high "resolved" rates.
• Review a regular sample of transcripts with a short rubric, checking facts against your real information.
• Track handoff reasons, wait times, context transfer and missed escalations.
• Combine customer feedback, repeat contacts and downstream data with the agent's own metrics.
• Set pause criteria in advance, and keep conversation data only as long as you need it.

Sources

• National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (January 2023)
• National Institute of Standards and Technology, NIST AI 600-1, Generative Artificial Intelligence Profile (July 2024)
• Consumer Financial Protection Bureau, Chatbots in consumer finance (June 2023)
• Federal Trade Commission, Protecting Personal Information: A Guide for Business