How Monte Carlo’s agentic observability caught a $16K/day bug in our own revenue data

A custom SQL monitor fired a revenue alert. 45 minutes later, our Troubleshooting Agent and Claude Code had diagnosed, fixed, and shipped the fix—with documentation. Here’s exactly what happened.

It started the way most data incidents do: quietly.

A couple of months back, a custom SQL monitor in Monte Carlo triggered on our internal finance table — finance_consumption_revenue_explorer (CRE). The rule was simple: flag any customer with duplicate rows on the same date. Up until now, it had never fired before. There was no Slack ping from an analyst, or communication from an executive asking why the numbers looked off. It just came through as an alert with a customer ID, and a question worth answering.

What the alert had caught was a 64x row duplication for one of our enterprise customers, and the revenue overstatement it was generating was approximately $16,200 per day.

Troubleshooting Agent’s investigation

The first thing we did to investigate the source of the issue was run Monte Carlo’s own Troubleshooting Agent on it, one of our platform’s internal agents that, alongside others like the Operations and Monitoring Agents, automate a huge amount of observability and data engineering work.

This is the part that used to take a human analyst 15–30 minutes of cross-referencing tables, Slack threads, and Salesforce records. The Troubleshooting Agent returned a response in seconds.

What it discovered was that two Closed Won Salesforce opportunities — a Net New and an Expansion deal — were both mapping to the same metronome_customer_id. The duplication had been running for 379 consecutive days, with row counts having gone up 12,599 from baseline.

The Troubleshooting Agent also offered a hypothesis for the root cause: that the ROW_NUMBER() QUALIFY deduplication logic in the CRE model was probably failing to filter out the duplicate opportunity. It was a reasonable guess, though it turned out to be wrong. Even so, it pointed the investigation at the right 1,100-line dbt model, which is all we needed.

The Claude hand-off

From there, we handed it off to Claude Code, connected to Snowflake via MCP. The two tools have their own distinct jobs to be done in this example. The Troubleshooting Agent is built for fast triage — it pattern-matches against alert history, table metadata, and lineage to surface the most likely culprit. What it can’t do is read model source code or run targeted investigative SQL. Claude Code then picks up the task to read model source code and run targeted investigative SQL.

Claude’s first queries confirmed what the Troubleshooting Agent found: 64 byte-identical rows per day for this specific customer account verified via HASH(*). Then Claude did something harder — it traced how the duplication got there in the first place.

The row count hadn’t always been 64. It had escalated: 1 row, then 2, then 8, then 16, then 32, and finally 64 — each jump corresponding to a new contract period being added over the course of a year. By the time the alert fired, the problem had been silently compounding for 379 days.

The actual root cause turned out to be a single OR condition in the opp-to-metronome join on line 500 of the CRE model:

ON (a.metronome_customer_id = z.opportunity_metronome_customer_id
    OR ingest_aliases ILIKE '%' || z.opportunity_primary_account_uid || '%'
    OR ingest_aliases ILIKE '%' || z.opportunity_primary_account_uuid || '%')

This join was designed to handle a common scenario: not every Salesforce opportunity has a direct metronome_customer_id, so the model falls back to matching on workspace aliases when a direct ID isn’t present. That fallback logic is legitimate and necessary — it’s how 9 other customers in the CRE get matched correctly.

The problem is what happened a few weeks before the alert fired. An engineer had merged two of this specific customer’s Metronome records into one, consolidating their workspace aliases under a single customer entry. After the merge, that single customer’s ingest_aliases field contained both workspace UIDs. The OR ILIKE join then matched both the Net New and Expansion opportunities to the same Metronome customer — creating a cartesian product instead of a join.

The ROW_NUMBER() QUALIFY dedup the Troubleshooting Agent suspected was actually downstream of this problem. The bug was one step earlier, and it had been invisible until the alias merge finally pushed the row count high enough to trigger the monitor.

The fix was conceptually straightforward: replace the single OR-based join with a priority-ordered two-step match. Direct metronome_customer_id match goes first, always. The alias fallback only fires when no direct match exists.

That one change collapsed this customer’s rows from 64 to 2 and corrected a $16,191 daily revenue overstatement — while leaving all 9 legitimate alias-only customers untouched.

Two PRs followed — one for dev, one for prod. We sent a Slack to the reviewer and documented the incident and the recovery process. The total wall clock time for remediation, from alert to shipped fix and documentation, was about 45 minutes, while actual keyboard time was only about 10–15 minutes of prompts and review.

The Troubleshooting Agent handled the initial investigatory work, which was crucial. It was able to go from “an alert fired” to “here is the affected customer, here are the two opportunities, here is the duration” in seconds, rather than an hour or more of manual triage that would be required of a human analyst. And, although the Troubleshooting Agent’s root cause hypothesis was wrong, it was a productive wrong: it told us which model to read and which logic to scrutinize.

Claude Code handled the rest. Reading the source, running the targeted queries, tracing the escalation, finding the real root cause, writing and validating the fix, and cutting the PRs.

Neither tool alone would have gotten there as fast. The Troubleshooting Agent needed Claude Code to finish what it started, and Claude needed the agent’s starting point to know where to look.

I often think about the version of this story where the alert never fires, and the 64x duplication sits quietly in the revenue table for another year, compounding with every new commit period, and inflating reported numbers. In this scenario, the discovery happens when someone finally notices at a board meeting or during an audit, something we were able to avoid thanks to Monte Carlo’s own agentic tooling.

That’s what data observability is supposed to do.

Our promise: we will show you the product.