Token prices are falling, so why is your AI bill going up?

There’s a clear narrative sweeping the software world these days: AI is getting dramatically cheaper. Or, to be more specific, inference is getting cheaper. 

Various researchers have been quantifying this: 

Stanford’s AI Index, for example, tracked the cost of querying a GPT-3.5-equivalent model falling from $20.00 per million tokens in late 2022 to $0.07 by late 2024 — a 280-fold reduction. Epoch AI, tracking the same phenomenon across benchmarks, estimates LLM inference prices have been falling somewhere between 9x and 900x per year depending on the task.

By standard software economic logic, therefore, your monthly enterprise AI invoice should be plummeting.

Yet, when engineering and financial leads review enterprise cloud and AI bills at month-end, the exact opposite seems to be happening. 

Welcome to the AI efficiency paradox. 

Uber became the poster child for this phenomenon in May when the company announced it burned through its entire annual AI coding tools budget in four months, then responded by capping spend at $1,500 per employee per tool per month. 

Among FinOps practitioners, 98% now manage AI spend — up from 63% a year earlier and 31% two years ago. 

And now, according to the 2026 State of FinOps report, the single most-requested capability that doesn’t exist in their tooling today is “granular monitoring of AI spend: tokens, LLM requests and GPU utilization.”

That’s an entire discipline saying, in effect: we cannot see what AI is costing us. 

Based on both data and anecdote, however, we know it’s going up. 

The instinct is to read the gap between falling prices and rising bills as waste – that someone isn’t optimizing usage, or using an expensive model when the cheap one would have worked, or even leaving a dev environment running somewhere in perpetuity. 

While there is certainly a level of those factors going on, the root cause of why AI costs are ballooning is structural rather than based on individual behaviors. 

Below I’ll explore some of the key reasons why enterprises keep seeing the bill go up while the unit price goes down and what we can all do about it. 

As unit costs decreased, the number of units increased

Think back to the days of purely generative AI when everyone became obsessed with asking ChatGPT to spell the word “strawberry.” 

That world, which seems so far away from us now, represents an enormously different usage pattern of AI vs. what has become commonplace in today’s enterprise. 

A chatbot query, after all, is only one inference call. You send a prompt, you get a response, and pay for the tokens in both directions. This is a clean, legible unit of consumption. It’s the unit almost every 2024-era AI business case was built on.

An agentic task, which is the prevailing use case of AI today, is so far beyond that. 

An agent calls tools, reads the results, evaluates whether it’s done, and decides what to do next. Each of those steps is a model call, and each call re-sends the accumulated context.

Researchers at Stanford’s Digital Economy Lab put a number on the agent AI tax in a paper published this year. Studying agentic coding tasks, they found them “uniquely expensive, consuming 1000x more tokens than code reasoning and code chat.” 

The cost sits almost entirely on the input side, not the output. 

An agent doesn’t write more than a chatbot does, but it re-reads far more: the original prompt, then the prompt plus its own response, then all of that plus the next response, compounding at every step. The paper’s term for it is a “pricey context snowball,” which is about as good a two-word summary of agentic cost structure as exists.

Whatever multiple applies to your workload — and it varies enormously by task — the pattern tends to be the same. 

So, to conclude: we’ve seen the price per token fall by a large factor, yet simultaneously tokens per task rose by an even bigger one. 

Predicting what a task will cost is impossible, even by the agent itself

We all know that AI is non-deterministic in nature; an agent can take many different paths to complete a task and generate an output. 

It turns out, however, that we’ve been underestimating the real weight of what it means to be non-deterministic. 

The same Stanford paper that found that agents consume 1000x more tokens than code reasoning and code chat sheds some light on how difficult it is to predict the true cost of tasks.

They found that, even when running the same agent on the same task, costs varied by up to 30x between runs. This is between identical invocations of identical work. 

Agent trajectories are stochastic; the agent cannot know in advance how much context it will accumulate, so it cannot know what it will spend.

What’s interesting is that the agent can’t even ballpark it very well; the researchers found models consistently underestimate their own token spend. As Stanford research scientist Jiaxin Pei put it:

“Agents are not capable of predicting their own token costs. This is the fundamental bottleneck for result-based pricing for agents. You can’t really price the agent well unless you can figure out the cost, but now you only see the token costs after everything is done.”

Consider the forecasting implications for this. 

A chat product’s API bill can be modeled from daily active users and average conversation length, but an agent fleet’s bill can’t be modeled that way at all since there is so much variance across single task executions. 

Cheap tokens change engineering behavior

There’s a second-order effect that comes into play here, more psychological than technological: the Jevons paradox. It stipulates that when the unit cost of a resource falls, consumption expands faster than the price drops, leading to total spend rising. While William Jevons was writing about coal in 1865, you can see how the theory applies to tokens. 

Back when inference was expensive, engineers economized without being asked. They trimmed context, used the smallest model that worked, and thought critically about whether a step really needed AI.

Nobody is doing that at a tenth of a cent per thousand input tokens. 

Instead, context windows get stuffed rather than curated, retrieval gets generous rather than precise, and multi-agent architectures become the de facto choice – because, why not? The coordination overhead doesn’t feel like it costs anything.

A sub-factor of this is the creep of what you might call “architectural laziness”: reaching for a model where ordinary code belongs. Production systems now routinely call an LLM to parse a date, reshape a JSON payload, do arithmetic, or match a string — operations that deterministic code performs instantly, correctly, and for nothing. 

A model is a probabilistic engine, not a general-purpose runtime. Every one of those substitutions buys you a permanent token tax, added latency, and a non-deterministic answer to a question that had a deterministic one.

You are billed for tokens you can’t really see

There are a couple of invisible line items that sit between the model price list and what you actually see on your invoice.

The first is reasoning. Modern reasoning models generate long internal chain-of-thought before producing a single word of user-facing output. Those “scratchpad” tokens are necessary because that’s how models can handle multi-step logic, and you pay for every one of them. 

The second is the scaffolding that enterprises wrap around the model to account for the fact that it is probabilistic. This includes monitoring and checks, like LLM-as-a-judge evaluation, where a frontier model scores the output of a cheaper one. Other examples of this ancillary infrastructure  are guardrail prompts, schema validators, and safety classifiers. While each is defensible for an organization that has to ensure output quality, it’s still another source of inference cost. A single user-facing query in a mature enterprise deployment, for example, routinely triggers several underlying calls. None of that multiplier appears anywhere in the vendor’s price-per-million-tokens.

There’s a supply-side version of this, too. Newer models often consume more tokens to finish the same task than the models they replace. This is by design: they reason longer and take more turns before answering, and it means per-token price and per-task cost can move in opposite directions. A model that launches cheaper per token can still cost more per completed job. If you upgrade and your bill goes up, it might just be that you bought more thinking per question.

Paying for capacity does not always get you capacity

There’s another cost factor here that has less to do with pricing than with scarcity. We’ve heard it most clearly from teams managing data pipelines, but the same mechanics govern provisioned inference.

A data architect at one of our customers described it to us this way: “Primarily the issue is compute unavailability, like the region we are in Azure, sometimes in East or West US… We run into the soft quota limit.”

His team had already bought their way out of this. Or thought they had:

“We already have pre-purchased compute. We still run into the compute unavailability issue, because purchasing compute on Azure doesn’t guarantee that compute is given to us. It just increases the soft quota for the region.”

He is describing his cloud provider’s documented behavior accurately, and the same distinction applies directly to inference capacity. Microsoft’s own documentation on provisioned throughput separates the two concepts: quota is a policy limit enforced per subscription and per region, carrying no cost; capacity is the actual resource available to be deployed, allocated at deployment time and held for the deployment’s lifetime. 

Buying provisioned throughput raises your ceiling, but it does not install a floor. Azure’s own guidance is to hold quota in more than one acceptable region — which is another way of saying the vendor expects you to plan for facing potential resource limitations.

This is why “we pre-purchased capacity” is not the risk mitigation a business case usually treats it as. It’s a reservation against a policy limit, not against hardware, and it applies as much to a provisioned inference deployment as to a Databricks cluster.

So what does a competent engineer do when the platform can decline to give them compute mid-run? They build a workaround:

“What we have done in the past is we have kept the timeout pretty high, so that if the job runs every hour, we keep the timeout 24 hours — then for sure, it will never be killed.”

An hourly job with a 24-hour timeout sounds like a reasonable piece of defensive engineering; it’s stopping the scheduler from killing work that’s merely waiting on capacity rather than actually stuck. 

But look at what gets traded away: a timeout is a circuit breaker, so the engineer hasn’t reserved 24 hours of compute; they’ve removed the thing that would have stopped a job that hangs. A wedged dependency or an agent stuck in a retry loop now burns for a full day before anyone intervenes. A false positive is traded for a far more expensive false negative.

A decision like this isn’t made in a planning meeting. It is the defensive (and understandable) measure of an engineer who has to stop their pipeline from dying.

Broken data has a price, too

This brings us to the section that is particularly important for teams running agents on top of  data infrastructure.

We are familiar with what happens when a traditional pipeline hits bad data: it fails, typically loudly, cheaply, and — if you’re instrumented well — quickly. The cost of a data incident has historically been counted in engineering hours, downstream trust, and the occasional wrong number in a finance deck somewhere.

But agents don’t fail that way. 

An agent that pulls from incorrect data does not stop. It re-reads the context, reasons about the discrepancy, tries a different approach, calls another tool, re-reads a larger context, and tries again.

Agents are designed to be resilient to ambiguity, which means that their response to bad input is to consume more tokens in an effort to complete the task, rather than just fail and stop consuming. 

Put that next to Stanford’s 30x same-task variance and it begs a more critical look at how data issues in your own environment are influencing cost. A meaningful share of the expensive tail of your token distribution is probably made of your data reliability problems. It’s the agent runs that go wrong and keep trying that blow past the median.

This suggests that data quality has acquired a unit price insofar as it feeds agents. This is useful because it converts a historically soft argument into a hard one – e.g. “failed and retried agent runs are a fifth of our inference spend.”

Strategies to reduce token spend

Control over token spend requires visibility and proper measurement.  Per-agent, per-task unit economics has to become a first-class engineering metric — instrumented, alerted on, and owned by the team that builds the workflow. 

Concretely, this means that teams must be able to: 

Measure cost per completed task. Monthly spend is a misleading number because it tends to just keep going up. It rises when you’re succeeding and rises when you’re failing for reasons we outlined in the sections above. It can’t distinguish, but cost per successfully completed task can.

Attribute spend to the workflow. If you can’t say what your document-classification agent costs versus your reconciliation agent, you can’t make an informed decision about either. Most organizations still attribute at the API key or subscription level, which is roughly as useful as a phone bill with no itemization.

Watch the distribution. Given 30x variance on identical tasks, the average cost per task is not the most insightful number. The p95 and p99 are where the money is, and those runs almost always have a specific, findable cause driving their outlier status. 

Cap the loop. Establish hard limits on retries, step counts, and accumulated context size. An agent with no ceiling is an unbounded liability.

Audit your pipelines for work that isn’t a model’s job. Every LLM call doing parsing, formatting, or arithmetic is a recurring charge for something that deterministic code does freely and correctly. This is the cheapest optimization available.

Track resource capacity availability as you would track uptime. With some compute providers, you may have bought a ceiling, not a floor. Whether you can actually get compute right now is a live reliability question that should be measured and alerted on. 

Connect data incidents to spend. When a table breaks, the critical question is no longer only “who’s affected downstream,” but rather “what did the agents do about it, and what did that cost.”

These processes are not obvious budgeting exercises, which is why they often do not happen. 

This is instrumentation work, and it is in the realm of the engineers who own the pipelines and the agents — the same people whose managers have started asking them, without much tooling to help, whether the spend went up or down after the outage.

The telemetry is there to help us reconcile growing costs, but most of us are throwing it away.