Debug Problematic Agent Behavior 4x Faster with Agent Observability

Needle Meet Haystack

An agent is a complex data + AI system with many interdependent, evolving components such as: input data, orchestration code, instructional prompts, foundational models, tool calls (external systems), and more. 

Even small changes or unanticipated edge cases can lead to poor operational performance and output quality. 

As AI pilots move toward production scale, engineers and data scientists struggle to troubleshoot these reliability issues that ultimately destroy trust, adoption, and ROI. Debugging typically involves hopping across multiple systems and manually combing through thousands of piecemeal, semi-structured LLM call logs.

Full Visibility & Advanced Filtering

Agent Observability treats data + AI as a single system allowing teams to easily monitor the reliability and quality of their agent inputs and outputs within a single piece of glass.

“…Combining data and agent observability in a single platform gives us visibility into the full agent lifecycle… all in one place.”— Travis Lawrence, Sr. Manager, ML Engineering at Pilot Flying J

The Trace Explorer feature provides end-to-end trace visibility to identify and debug undesired agent behavior. Every agent run is mapped with an intuitive, easy to navigate tree structure clearly outlining each step taken by the agent along with metadata for critical context.

Users can navigate each sequential span of a trace to see the associated  inputs, system prompts, completions, duration, token count, model version, model provider, and workflow description of each step. Every LLM and tool call is linked to a specific task, making agent behavior transparent and explainable.

This end-to-end visibility makes agent debugging more than 4x more efficient as teams can now quickly identify and drill into a problematic span to answer questions like:

• Is a task bottlenecking others? Did a tool or LLM call get stuck in a loop, timeout, or fail? 
• Was there an insufficient or excessive amount of input provided? Was the output fit for use?
• Was the system prompt or model configuration fit for this scenario? Have any changes been made?
• And more

Trace Explorer also allows users to filter traces by latency, token consumption, model version, and workflow to identify recurring issues. For example, an engineer may want to review all traces with anonymously high latency, token counts, or those associated with a specific task.

How We Made Our Customer Facing Agent More Reliable

We use Agent Observability to monitor the reliability of the AI agents within our own platform. One agent we monitor, Troubleshooting Agent, automatically identifies the root cause of a data quality issue in about 30 seconds. 

Our lead data scientist, Elor Arieli, sets agent monitors and periodically reviews its performance to ensure this customer facing feature remains reliable and trusted. 

Using Trace Explorer, Elor isolated and examined the traces of the 10% slowest runs of the Troubleshooting Agent in the last week, which were taking more than 4 minutes to complete. 

Below we can see most of the agent’s tasks before making its first LLM call were performed in less than a second. However, the “grab_upstream_field_anomaly.task” took 77% of the total runtime. This is all before the first call and a likely bottleneck.

Compare this to the “pre_summary_reasoning,” task, which took more than 30 seconds, but on the other hand is a LLM call processing a considerable amount of information and thus some latency is expected.

When he explored further he quickly noticed a few things:

• All of the traces related to the same customer;

• Most of the abnormally long spans (but not all) were related to the same task; and

• All of the outputs of the abnormally long spans contained an identical unexpected response. “No access to [redacted].” Essentially, the agent was struggling to complete the task because this particular customer had turned off a necessary permission and the agent was attempting to continue its task in a futile loop.

This diagnosis, which could otherwise have taken half a day or longer, was completed in about 15 minutes. 

Data + AI In A Single Pane Of Glass

Monte Carlo’s Agent Observability extends end-to-end tracing with production-grade monitoring and evaluations across data and AI. 

Unlike point solutions, Monte Carlo closes the data + AI loop and stores all telemetry securely in your environment, giving teams full control, governance, and scalability from day one.

If you’re building agents that matter, observability can’t be an afterthought. Schedule a demo to see how leading teams are monitoring, debugging, and operating AI agents with confidence at scale.