Trust your Snowflake Cortex Agent Output with Conversation Clustering in Monte Carlo

Snowflake Intelligence has made it dramatically easier to put a natural-language interface in front of your data. A Cortex agent gets stood up, pointed at a semantic view, and handed to a team — and suddenly hundreds of people who never wrote a line of SQL are querying governed data by just typing a question.

That’s the promise, but it’s also a challenge.

Once real users get their hands on a Cortex agent, the questions they ask rarely match the questions you imagined when you built it. An analyst may have deployed a finance agent, but half the traffic is procurement asking about vendor spend. Or perhaps a team built an agent for one semantic view, yet users keep pushing it toward tables it was never scoped for. The agent is doing something all day, every day — but what, exactly, and for whom?

The individual conversations are all there. Snowflake Intelligence logs rich trace data natively, including the full conversation thread, the SQL each turn generated, the tool calls, the token counts, and the run’s status. Monte Carlo reads that trace data directly from your warehouse, so every Cortex conversation is available to inspect turn by turn. Inspecting conversations one at a time may answer the question of what happened in a single thread, but it doesn’t provide any insight into the overall performance, health, and patterns of the agent in aggregate.

That’s what conversation clustering is for.

From thousands of conversations to a handful of topics

Monte Carlo supports conversation clustering for your Cortex agents. The Clusters panel in the Conversations tab groups your Cortex agent’s conversations into named topics, with each cluster coming with a plain-language description of the intent behind it, its share of total traffic, and a conversation count. One click filters the conversation list down to just the threads in that cluster.

Instead of scrolling a feed of individual conversations, you open the tab and immediately see the shape of demand. 

The topics are derived from what users actually typed, not from a taxonomy you had to define in advance, and new conversations get sorted into topics as they come in, so the view stays current as usage evolves.

This gives you a top-down map of your agent that trace data alone can’t provide.

What that map is good for

Confirming the agent is used for what you built it for. The fastest signal in the Clusters panel is whether the topics match your intent. A Cortex agent scoped to sales analytics whose largest cluster is a support-adjacent topic isn’t broken. Rather, it’s telling you where real demand is, and where your next semantic view or verified query should go. Share-of-traffic turns a vague sense of “people seem to like it” into a ranked list of what people actually do with it.

Finding the gaps your semantic model doesn’t cover. Cortex Analyst is only as good as the semantic views behind it. Clusters make coverage gaps visible in aggregate: a topic that shows up with meaningful volume but weak answer relevance is a concentrated pointer to questions your semantic model handles poorly. Rather than guessing which synonyms, metrics, or verified queries to add next, you can prioritize by the size of the cluster that’s underperforming. A large “uncategorized” share is its own signal: it usually means users are asking things the agent wasn’t designed to field at all.

Prioritizing where quality matters most. Not every topic deserves equal scrutiny. Clustering lets you sample representative conversations from your highest-volume topics and run evaluations where they’ll move the needle, instead of spot-checking a random feed. When you’re deciding where to point Monte Carlo’s agent evaluation monitors, “the topic that’s 33% of traffic” is a better answer than “whatever’s most recent.”

Attributing cost to what people ask. Cortex agents can be expensive in ways that aren’t obvious from a monthly credit line. Because the conversation list carries duration and token counts alongside the cluster label, you can see which topics are driving consumption. A single analytically heavy cluster doubling its token footprint is exactly the kind of pattern that shows up early in a Cortex deployment. Clustering turns “usage went up” into “this category of question went up.”

Enabling the next wave of users. The topics your current users gravitate toward are the best possible starting point for documentation, example questions, and onboarding. What are people actually asking a Cortex agent to do? The Clusters panel is the answer, backed by real traffic rather than assumptions.

Zero instrumentation, warehouse-native

None of this asks anything of your agents. Cortex logs its trace data to Snowflake natively; Monte Carlo reads it directly, classifies conversations into topics, and keeps the view current as new conversations arrive. There are no code changes, no SDK, and no additional infrastructure required. And because Monte Carlo’s Agent Observability is warehouse-native and available on the Snowflake Marketplace as a native app, the prompts, completions, and conversation content stay inside your Snowflake environment, under the governance and access controls you already run.

The result is a top-down view of every Cortex agent you operate: not just is it healthy, but what is it for, who’s using it, and where should it get better next, all answered from the conversations themselves.

Curious what your Cortex agents are actually handling? See Agent Observability for Snowflake Intelligence →