Agent Trust in 2 Weeks: Part 1, Instrumentation (that’s not so hard anymore)

This post is part of Your First Two Weeks of Agent Trust, a four-part series on what it actually takes to trust an agent running in production. Trust here isn’t a feeling, it’s a set of conditions that either hold or they don’t. Each blog works through one critical step to achieving this: getting traces flowing, deciding what’s worth monitoring, building the monitors that matter, and then running the whole thing against your own agent, one trust pillar at a time.

Agent observability projects often stall because the up-front work to set it up is deemed as too burdensome. Teams estimate that they would need a few weeks of engineering time to instrument everything, configure monitors, and actually be collecting data.

That estimate used to be fairly accurate. Standing up agent tracing meant stitching together OpenTelemetry SDK documentation, collector configuration, warehouse ingestion, and table registration — and then hoping traces actually showed up on the other end. This type of work required a fair amount of observability expertise and often took at least a week.

Fortunately, this type of timeline is not what teams using Monte Carlo have to work with. Depending on how your agent is built, there may be nothing to instrument at all — and where instrumentation is still required, you’re no longer the one writing it line-by-line. This post covers how to tell which situation you’re in and what each instrumentation scenario involves. It also flags a few common points that trip engineering teams up and how to deal with them.

The first question: platform agent or custom agent?

Before you estimate anything, work out which category your agent falls into, because the answer changes the amount of effort involved in setting up traces in Monte Carlo:

If you’re on a managed platform: little or nothing to instrument

For agents built on Snowflake or Databricks managed platforms, the platform is already producing trace data. Monte Carlo reads it from where it already lives, through the same connection you set up for data observability. No SDK, no collector, no pipeline, and no new credentials. The amount of work left over does vary between them, however.

Snowflake Cortex Agents are the genuine zero-touch case. There is nothing to deploy and nothing to instrument. You grant your existing Monte Carlo service account the right privileges, then go to Settings → Agent Observability, toggle to Platform Agent, and select your warehouse and agent. Monte Carlo auto-discovers the Cortex Agents available to it from there.

Databricks Agent Bricks and Knowledge Assistants need no SDK, but they aren’t quite zero either. Databricks logs traces automatically, but those traces only reach Unity Catalog once you’ve enabled trace storage — until you do, Monte Carlo will discover the agent and then flag it as having no trace table. That’s a single setting rather than an engineering project. If you’re running code-based custom agents on MLflow, there’s a minimum version requirement to check as well.

Genie works differently again. Genie conversation history lives behind an API rather than in Unity Catalog, so Monte Carlo installs a managed collector job into your Databricks workspace. About once an hour it reads new conversations and merges them into a Unity Catalog table. You don’t deploy it, upgrade it, or run it — Monte Carlo manages its lifecycle. The thing to note is the cadence rather than the effort: Genie data arrives hourly, so it isn’t a real-time view.

If you built your own agent

Custom agents still need to be instrumented. What’s nice is that you’re no longer the one doing it by hand.

The MC Agent Toolkit is a bundle of skills, hooks, and an MCP server that teaches your coding agent when and how to reach for Monte Carlo. It includes an instrument-agent skill that runs from your code editor. From a single prompt, it detects the AI libraries in use in your codebase, installs the Monte Carlo OpenTelemetry SDK at compatible versions, places the decorators that categorize your LLM calls for evaluation, and then verifies that traces are actually landing. It proposes its changes file by file and waits for your approval on each one, so nothing lands in your repository that you haven’t read. This is the element that turns what used to be a multi-day SDK exercise into a single prompt.

One honest caveat about version compatibility: some third-party OpenTelemetry instrumentation packages have to be held at specific versions to emit attributes in the format Monte Carlo expects.

The infrastructure side has been simplified, as well. The trace platform deploys as one Terraform module per cloud — AWS (EKS), Azure (AKS), or GCP (GKE) — with the cloud-native equivalents swapped in for secrets management and ingress. The setup is three steps:

1. Apply the Terraform module in your own cloud account.
2. Add the resulting ClickHouse instance as an integration in Monte Carlo.
3. Point your agents at the new collector endpoint.

Setting expectations properly: the Terraform apply itself typically runs 15–25 minutes, most of that spent waiting on the cluster and node groups to come up. And if you’re deploying on GCP, there’s a prerequisite worth starting early — Vertex AI model access approval, which can take 24 to 48 hours. Neither is a week of engineering, but neither is instant, and it’s better to plan around the real numbers.

If you were previously blocked because you don’t run on AWS, that constraint is gone. The same architecture now deploys on Azure and GCP.

Where teams still get stuck

As with any type of instrumentation, some parts can be a bit fiddly. Here are a few places where engineering teams get stuck and how to combat them, so you can check for these before you start rather than after.

Your agent code needs to be Python. This is the qualifier that most often ends the conversation early. If your agent is written in Go, Java, or anything else, the question becomes whether you have a Python orchestration layer that could carry the instrumentation instead.

Cortex permissions are granted per agent, not per role. The MONITOR privilege has to be granted on each individual Cortex agent object — a blanket role grant looks like it worked but isn’t sufficient, and agent discovery then fails silently. If agents aren’t appearing, this is the first thing to check, and running SHOW AGENTS in Snowflake will confirm what’s actually visible before you go digging anywhere else.

Snowflake redacts prompts and completions by default. Since Snowflake’s April 2026 behavior change, prompt and completion content comes back redacted even when your permissions are otherwise complete. There’s an account-level grant that resolves this. It isn’t a bug and it isn’t a permissions gap on your side, but it does surprise people mid-demo, so handle it up front.

Databricks traces have to exist before you register the agent. If you add the agent in Monte Carlo before any traces have been written to Unity Catalog, registration fails. Confirm you have recent rows first. Relatedly, Monte Carlo reads a table prefix rather than a literal table name, so a hand-built export table with a similar-looking name will never be picked up no matter how much good data is sitting in it.

Your agents need a private network path to the collector. The collector endpoint sits behind an internal load balancer with a private IP, so your agents have to reach it over private connectivity — same VPC, peering, VPN, or Interconnect — and their source range has to be on the allow-list. This is less a blocker than a networking conversation to have early, and it’s far easier now that you can deploy the platform in whichever cloud your agents already run in.

Common questions you’ll get asked

If you’re the person championing this, you will get both of these, so it’s useful to have the answers ready.

“Are we locking ourselves into a proprietary format?” No. Monte Carlo’s instrumentation is built on OpenTelemetry, which is the vendor-neutral standard the broader observability ecosystem has been using for years. Your traces conform to the emerging GenAI semantic conventions that the wider industry is converging on, which means they’re portable — if you already run OTel infrastructure, this telemetry can flow into it alongside everything else.

“Where do our prompts and completions actually live?” In your cloud account. The trace store is deployed inside your own AWS, Azure, or GCP environment, and Monte Carlo queries it at runtime to power the UI and evaluations. Prompt and completion content — which is where your PII and sensitive business context lives — stays under your existing access controls. For teams in highly regulated industries, this is an important consideration.

What you get before you build anything

You may be surprised to hear that you don’t have to build anything within the platform to get value on day one.

Once traces are flowing, you see your Agent Summary view automatically — a dashboard of trace volume, latency distributions, token consumption, and error rates. This requires no monitors, configuration, or thresholds to set from your side.

For many teams, simply being able to see what the agent is doing is the first real win, and it’s a reasonable place to stop and take stock before you start building anything deliberate.

Which brings up the question this series turns to next: now that you can see everything, what should you actually be watching?

Start here

You don’t have to work this out alone, and you shouldn’t have to. We run the first two weeks with you — one pillar at a time, on your agent, against your data.

Watch for the next blog in this series: Part 2 — What to Monitor First.