The Desperate Need For An “Agent Contract”

A system of systems

We were lucky enough to collaborate with Chad Sanderson and Andrew Jones on seminal works that helped popularize the “data contract” methodology.

The core problem this methodology sought to solve is exactly what is driving AI trust and reliability issues today: namely, the lack of alignment between teams and systems that are disparate but dependent. 

In the case of data contracts, the problem was data producers (often software engineers) changing their data outputs in ways that, when ingested, broke downstream data products managed by the data team.

In the case of agents, the problem is these AI systems ingest data or use tools managed by other teams. When these components change, or are not at the expected level of reliability, there is no model or engineering fancy enough to turn an unreliable input into a reliable output.

Data contracts are not a magic wand, but many systems have become more reliable for teams that explicitly defined the “handshake” between systems. 

This agent contract (or honestly whatever you want to call it, we’re not here to invent buzzwords) needs to be priority one for data + AI leaders.

The core principles of an “agent contract”

Here’s how we would propose to get started using core principles:

Work backwards

It never should have been a radical idea, but it was: gather requirements from your consumers and build towards them. 

The inputs for one system should not be tortured and twisted ad-hoc from the natural outputs of one intended for another purpose.

To ground this in the practical: a data product–whether dashboard, machine learning application, or AI agent–should be able to access governed, curated, and fit-for-use datasets. If not your data scientists and agents will be scraping, transforming, and interpreting data–make it simple for them!

The consequence as Chad has colorfully put it in many a LinkedIn post:

Agree to standards and definitions

You have a purpose built dataset and even a semantic layer that helps guide your agent to get exactly what it needs, no more, no less. Great! 

But what if those inputs are incorrect or change? 

For those that haven’t grasped it yet: your output reliability will never be greater than your input reliability. Your agent SLA will never be greater than your data SLA or the reliability of the tool being called.

This is where leadership needs to step in. Your AI engineer isn’t thinking about data quality. Your data engineer isn’t thinking about agent reliability. 

We need to build human relationships…and then codify that agreement in a contract (like marriage!). Otherwise schema changes break pipelines, changing definitions create drift, and tool calls fail.

The final piece is to create a process for change management including the steps for enforcing or amending the contract. 

What should be in an “agent contract”

Let’s dive on level deeper and imagine what an agent contract would contain. 

It would likely cover four different organizational groups mediated by governance and leadership teams: knowledge management, data engineering, AI engineering, and IT. 

Any changes to any of the below items would require notification and amending the contract (and ensuring system interoperability isn’t impacted).

Knowledge management- Unstructured data

• What files are referencable in which repositories 
• Classification (sensitive, PII, canonical) of those documents
• Master definition of key or industry specific concepts 
• The file type, structure, and length of documents (sections, bullets,  token count)
• Prohibited content patterns (screenshots of text, etc)
• Chunking strategy and metadata guarantees (required fields, IDs, timestamps)
• Retrieval method, indexing strategy, and metadata requirements
• Expected recall / precision targets for key document classes
• Versioning, deprecation, and supersession signaling
• Ownership, edit permissions, and escalation paths for disputes or breakage
• Change notification requirements for modified or replaced documents

Data engineering- Structured data (mostly the traditional data contract)

• What datasets will be referenced where
• The schema of the dataset
• The semantic meaning and lineage of key metrics
• Data freshness and latency
• Data quality SLAs across the 6 dimensions of data quality for key metrics (what is the downtime and average time to resolution)

IT- Tool availability and performance

• What tools or services are approved for agent access 
• Authentication and authorization mechanism used by agents (service accounts, scopes, RBAC)
• Rate limits, quotas, and concurrency constraints per tool
• Expected uptime and availability SLA for each tool
• Maintenance windows and change notification policy
• Security and compliance constraints (PII handling, data egress restrictions)

AI engineering- Prompt, model, calls

• Task and success/failure conditions associated with the (tool/data) call
• System prompt and model involved in the (tool/data) call
• Required inputs and outputs formats 
• Maximum steps, retries, chained calls
• Golden dataset for trajectory and output evaluation
• Alert thresholds for detecting regressions in production outputs
• Desired SLA tool call latency
• Versioning and rollout strategy for prompt or model changes

Buzzword or Helpful Framework?

You can call this an agent contract, a handshake, or just “doing the hard coordination work early.” The label doesn’t matter.

What does matter is recognizing that agent reliability is not primarily a modeling problem. It’s an organizational one. 

As agents become embedded in real business workflows, they inherit all the same failure modes we’ve already seen in coupled systems: silent breaking changes, unclear ownership, mismatched expectations, and downstream teams left to absorb the blast radius.

This is a lot of work, but no one said shipping reliable agents was going to be easy. What are we missing? Let us know.

Our promise: we will show you the product.