Optimizing agent-user interactions with Monte Carlo’s evals

When it comes to AI agents today that support user conversations, such as customer service or booking agents, the complexity of their conversations is impressive. A user might ask something, the agent responds, the user pushes back or clarifies, and the exchange goes back and forth until the task is done or the user gives up.

This evolution in agentic conversations has changed how we think about performance and the questions around what “working correctly” even means. An agent can return technically valid responses at every turn yet still leave the user frustrated, confused, or without what they actually needed.

The engineering teams building these agents already know this. Performance monitoring for agentic conversations requires more nuance to address the complexity of these conversations, gauging things that are more qualitative in nature like politeness, clarity, and helpfulness. The challenge is now proving that conversational agents are behaving the way they are intended, at scale, across thousands of conversations, without reading every transcript by hand.

That’s what Agent Evaluation Monitors are for, and Monte Carlo has been shipping a steady stream of improvements that make conversation-level evals sharper, faster to act on, and easier to actually use day to day.

The rise of agent evals

Agent evals have become a standard way of automatically grading how well an AI agent is performing, instead of just checking whether it ran without errors. Rather than looking at a single response in isolation, evals can score a whole conversation, an individual LLM call, or a full multi-step workflow against specific criteria. These include criteria like whether the agent actually completed the task, whether its answer was relevant and helpful, whether it stayed on topic, or whether the user ended up frustrated.

Some evals use an LLM-as-judge to score against a rubric (e.g., a 1–5 quality or satisfaction score), while others are deterministic checks against the raw output (e.g., is this valid JSON, does it contain a required field).

The point is to catch quality problems that never show up as a technical error (an agent can complete every call successfully and still be giving bad answers), and to do it at scale across thousands of conversations rather than by reading transcripts one at a time.

Why grading a single response isn’t enough

Most eval adoption so far has focused on the easiest layer to grade: the individual span or LLM call. That’s a reasonable place to start, as spans are self-contained, so a rubric can score them in isolation. But an agent isn’t just a series of unrelated calls; it’s a back-and-forth with a user that unfolds over multiple turns. Grading each call on its own can tell you every response was technically fine and still miss the fact that the interaction, taken as a whole, did not play out as intended.

Determining whether an agent is actually working properly requires analyzing signals across an entire conversation, not within a single call or span. A user who has to re-explain themselves three times to an agent, ask for a human, or push back on a wrong answer is telling you something a single-span check will never catch.

Until recently, getting that visibility meant one of two things: reading conversations one at a time, or waiting for an eval alert to point you somewhere useful, and then digging through many spans to find the actual evidence.

As an aside, several teams at Monte Carlo actually independently hand-built their own “how frustrated was this user” eval before it became a default. This inspired us to make the capability a true Agent Trust platform feature, extending its availability to all of our customers.

Full conversation evals are now in the Monte Carlo platform

A number of recent releases work together to make conversation-level evaluation something you can act on, not just something you can compute:

• Entire-conversation evaluation — Agent Evaluation Monitors can score a full multi-turn transcript as a whole, not just individual spans, using built-in templates like task completion, helpfulness, clarity, and prompt adherence. This means you can measure outcomes that emerge only when you observe across an entire exchange, rather than obvious ones that surface in a single output.

• A built-in satisfaction/frustration score — a new Satisfaction preset scores every conversation 1–5 (5 = composed and constructive, 1 = extreme frustration) with a defined rubric, scoring anchors, and bias guardrails, so you no longer have to write that eval yourself.

• Eval scores inline on the conversation page — frustration score, task completion, and other eval results are shown directly on the conversation itself, so you can scan a conversation’s quality without leaving the page.

• Sample conversations right on the alert page — when a conversation eval alert fires, the low-scored, breached conversations that triggered it now show up directly on the alert, instead of requiring a trip through “Investigate” and “view sample conversations” to find them.
• Conversations clustered into topics — a new Clusters panel automatically groups a workflow’s conversations into named topics, each with a description and share of traffic, so you can see what users are actually asking your agent and jump straight from a topic to the underlying conversations, without having to know what to search for in advance.

• Customer account shown automatically — the conversation detail page now resolves and displays the customer account directly from trace metadata, with no setup required, so you can scope quality and debugging to the right customer while triaging.

Put together, the workflow looks less like spot-checking transcripts and more like an actual quality feedback loop. You can see the topics your agent handles, know which ones are trending unhappy, jump straight to the conversations that prove it, and view the score without ever leaving the page.

Get started

If you’re running agents in production and want to see how they’re actually landing with users — not just whether the calls succeed — Agent Evaluation Monitors are now a core part of Monte Carlo’s Agent Trust Platform. Schedule a demo to see it on your own agents.

Our promise: we will show you the product.