Scaling Agent Bricks trust: from traces in Delta tables to production-grade observability with Monte Carlo 

Databricks writes Agent Bricks trace data to your own Delta tables in an open, portable format. With Monte Carlo, you can monitor, evaluate, and use this telemetry for agent optimization at production scale. 

The advantages of keeping agent traces in the Databricks ecosystem

Most agent observability tools are designed for you to forward the traces you collect somewhere else. Databricks’ Agent Bricks tooling is different and more convenient: trace data is captured through MLflow Tracing in the OpenTelemetry (OTel) format and written directly to Unity Catalog Delta tables that you already own, queryable through any Databricks SQL warehouse. 

Databricks itself frames the case for this as coming down to three things: 

• Storing high-volume trace data in Delta on object storage tends to be cheaper than SaaS-based retention pricing
• Keeping raw prompts inside Unity Catalog avoids the governance friction of sending sensitive data to a third party
• Once traces are tables, they can be joined with the rest of your business data rather than living in an isolated observability silo 

These are real advantages, and they serve as the basis for an excellent starting point when data and AI teams want to interrogate agent performance and behavior. Let’s start here before discussing how Monte Carlo complements this process and levels it up for teams looking for production-grade agent trust infrastructure. 

What does a raw Agent Bricks trace actually look like sitting in your own Delta tables, and what can you do with it using SQL queries?

What’s in the tables

By default, MLflow traces live in MLflow’s own internal tracking store, which is not something you can query with SQL. Databricks lets you point an experiment (the container MLflow uses to group traces for a given agent or app) at Unity Catalog instead. This is a one-time setup step where you bind the experiment to a catalog, schema, and table prefix. 

Once that’s done, Databricks provisions a set of Delta tables and views under that prefix: 

• Separate tables for OTel spans, logs, and metrics
• An MLflow-specific annotations table for tags, assessments, and feedback
• Consolidated views: a trace_unified view that assembles all of it into one record per trace, and a lighter trace_metadata view for just the MLflow-specific fields. 

Every agent interaction becomes a trace made up of nested spans: a top-level AGENT span for the full conversation turn, wrapping CHAIN spans for planning steps, LLM/CHAT_MODEL spans for individual model calls, and TOOL or RETRIEVER spans for anything the agent called out to.

You can query this directly:

SELECT
  trace_info.request_time,
  trace_info.execution_duration_ms,
  trace_info.state,
  spans
FROM catalog.schema.table_prefix_trace_unified
WHERE trace_info.request_time >= DATEADD(day, -1, CURRENT_TIMESTAMP())

You will see that spans come back as nested JSON: arrays of span objects, each with its own attributes, timing, and status buried a few levels deep. A single trace might contain a dozen spans across four or five types. To get anything resembling “average tokens per agent, per day” or “error rate by span type,” you’re writing recursive LATERAL VIEW EXPLODEs against JSON, extracting span_type, status, and token attributes out of unstructured fields, and re-aggregating by hand every time you want to ask a new question at that kind of rolled up view.

This is not a criticism of the format. Far from it; raw, structured, queryable Delta tables are exactly what you want as a foundation. But, depending on what kind of insight you want to get, and at what scale, “queryable” is not always “operationally useful.”

The limitations of SQL querying against Delta tables

Ad hoc queries get you moment-in-time snapshots. You can easily see today’s error rate or this week’s mean token count. Databricks’ own documentation on this is telling: native dashboards cover trace volume, errors, latency, token usage, and cost. For most teams, that’s enough for day-to-day monitoring. Anything past that means writing and maintaining custom SQL yourself. Databricks’ own published example shows how they built a custom “Tool Performance” widget, breaking down latency and error rate per individual tool, and a separate “Custom Cost Analysis” dashboard for accurate pricing. 

What that hand-built layer typically has to solve for:

• A rolling baseline. Understanding what “normal” looks like regarding an agent’s token usage or a span duration, for example, requires knowing what “normal” has historically looked like, adjusted for the fact that usage is naturally spiky.
• Cross-span correlation. Different combinations of signals across span types point to different root causes, even when each signal alone looks similar. Take, for example, rising token counts on CHAIN spans. If this is combined with a falling completion rate on TOOL spans, it suggests the agent is engaged in more elaborate planning because its tools are failing and it has to compensate. This points to a tool/infrastructure problem. In another example, say that you have flat, normal token counts combined with declining eval scores. This suggests the agent is running fine mechanically but producing worse answers, which points to a model or prompt problem, not an infrastructure one. Both examples indicate that something is off from a distance, but they point in opposite directions when it comes to actual diagnostic steps. Revealing either pattern means writing a query that explodes both span types, aggregates each by trace or time window, and joins them back together. This requires filtering to one span type at a time, which is what a simple query naturally does, and can only ever show you half the picture.
• Alerting with a destination. A SQL query that returns a concerning number is only useful if someone is looking at it. Turning that into “Slack the on-call owner with the specific trace IDs” is a completely separate build.
• Accurate cost. Databricks’ own team ran into this directly: their native cost metrics rely on standard list prices, which are wrong for any team with negotiated rates or fine-tuned models. In their own published example, the fix was building a custom dashboard with pricing logic hand-embedded into the SQL query to get an accurate cost-per-trace number. While that’s a working fix, it’s still a one-off query someone has to build and maintain per team, rather than something that comes with the platform by default.

These are not limitations of the trace data; it all exists. Rather, it becomes a question of what it takes to operationalize it continuously, across a growing fleet of agents, without every team building their own version of the same aggregation pipeline. 

Ultimately, this is where DIY-ing solutions on top of Delta tables hits a usefulness ceiling, and an expert solution is required. 

How Monte Carlo complements this for scaling agents

Monte Carlo connects directly to the same Unity Catalog Delta tables Databricks already writes to. This means you can get the same OTel-formatted traces and the same span-level data, with no new instrumentation, no SDK, and nothing new to deploy. What changes is what happens after the connection:

• Span types become metrics, automatically. Instead of writing a new EXPLODE query every time, AGENT, CHAIN, LLM, TOOL, and RETRIEVER spans are pre-aggregated into token count, duration, and status by type. So, “mean tokens on CHAIN spans, last 7 days” is a monitor that you can deploy (even doing so autonomously with our Agentic Operations, if you wish), not a query you write from scratch.

• Anomaly detection replaces manual baselining. Instead of eyeballing whether today’s number looks high, ML-driven monitors learn what normal looks like per agent — including tolerating the natural spikiness of business-hours-heavy traffic — and flag real deviations in mean tokens, P50/P90 duration, or error rate.

• Signals combine instead of living in separate queries to surface real insight, fast. A combination of high token counts on planning spans next to a falling tool-call completion rate will surface as one incident in Monte Carlo, not two unrelated numbers that you would have to intentionally analyze to tease out a relationship between them.

• Cost reflects what you actually pay. Negotiated rates and fine-tuned model costs get factored in directly, instead of every team building its own pricing overlay on top of list-price defaults.

• Alerts go somewhere. Anomalies route to Slack, Microsoft Teams, PagerDuty, or ServiceNow with the specific trace IDs attached, with a tracked resolution path. Engineers can retrace a specific error on the exact run that experienced it, instead of having to remember to check a query result on a regular basis.

Two patterns this catches in practice

Having this level of granular visibility, supported at scale, means that your teams can feel confident about deploying agents at scale. Here are a couple of examples of common performance patterns that get picked up very quickly when you have Monte Carlo layers on top of Databricks, rather than relying on regular querying. 

Token spikes. A pattern Monte Carlo sees often in the early weeks of an agent’s deployment is this: mean token consumption roughly doubles within a few days. The same symptom (tokens went up) can mean a few different things, however, and each one requires a different fix:

• Sudden spike, partial self-correction. Tokens jump, then drift back down partway on their own. This is usually temporary: a single high-token user session pulled up the daily average, or a system prompt or tool config was changed and then reverted. Often there is nothing to fix beyond confirming it was a one-off.
• Step-change with no correction. Tokens jump and then stay at the new, higher level indefinitely. Here, something has changed and is still in effect. It could be a permanent prompt edit, a new version of the calling application passing more context with every request, or a retrieval setting now pulling back larger result sets. The fix requires finding — and deciding whether to revert — that specific change.
• Gradual drift upward. No sharp jump, but a slow climb over days or weeks. This is usually structural. Perhaps, conversation history isn’t being trimmed, so every turn re-sends the full prior conversation and the token count compounds as sessions get longer. The fix is architectural; in this case, adding context window truncation.

Telling these three apart means looking at more than the token count itself. For the window where tokens moved, you need to check whether individual inputs got longer, whether the agent took more planning steps per conversation, and whether tool-calling behavior changed. Those three signals are captured by different span types: input length and model calls on LLM spans, planning steps on CHAIN spans, and tool behavior on TOOL spans. Diagnosing the pattern, therefore, means comparing across span types for the same time window, not just watching one number climb. 

This is a slow process to reconstruct by hand each time a spike happens, but it’s fast when monitors are already tracking each span type’s baseline and can surface the comparison automatically.

Usage volatility. Early agent deployments, especially internal tools and B2B workflows, rarely get steady, even traffic throughout the day. Usage clusters around business hours, specific workflows, or particular teams. Instead of a consistent volume around the clock, a day might see hundreds of requests in a four-hour window and almost none overnight. That unevenness has an easy-to-miss consequence for monitoring.

Most anomaly detection works off an aggregate number – say, for example, average error rate for the day. The detection system flags it when the number drifts far enough from its recent baseline. On a low-traffic day, however, that aggregate is built from very few data points. If only 20 requests come through instead of the usual 400, and 4 of them fail, that’s a 20% error rate. However, 4 failures is a small enough count that it may not move the day’s aggregate past whatever range the model has learned to treat as normal, especially since the baseline was already built to tolerate the natural day-to-day swings of spiky traffic. The failures are real; they’re just statistically hiding inside a small, quiet sample.

The fix isn’t a more sensitive volume-based monitor; that would just start firing on ordinary busy days that are actually fine. It’s tracking error rate as its own dedicated signal, with its own baseline tuned for uneven traffic, independent of overall volume. That way a bad day with low traffic can still get flagged on its own terms, instead of needing to move a shared, volume-weighted metric that’s built to expect noise.

Maintaining agent trust at scale

Databricks provides incredibly useful tools for exploring and troubleshooting agent performance issues within the same platform where you’re building and storing your data. Being able to query agent traces in your Delta tables via SQL is extremely powerful. 

However, there are different requirements when the task at hand is to build, deploy, and safely manage an agent fleet that holds up in production. 

Your monitoring and troubleshooting process has to withstand the operational pressures that show up at scale. We’ve discussed some of these examples in this blog: traffic that’s naturally uneven, symptoms that look identical on the surface but have opposite root causes, cost figures that quietly drift from list price, and spikes that need to be triaged in minutes rather than discovered days later in a dashboard. 

If you’re serious about scaling agentic systems in a way that maintains trust in what they produce and the impact they have across your organization, that operational resilience is essential.

Monte Carlo’s Agent Observability connects directly to the Unity Catalog Delta tables Databricks already writes MLflow traces to — no new instrumentation required — turning open trace data into continuous monitoring, owned incidents, and one unified view spanning both the data layer and the agent layer.

Our promise: we will show you the product.