We Used to Optimize Code. Now We're Told to Just… Spend More?

There's a phrase that's been circulating since this year's big AI hardware keynotes and some version of it has become the mood of the entire industry: use more tokens. Let the model reason longer. Spend freely, that's where the gains are.

I heard it and I nearly put down my coffee.

I have spent my career writing code that has to work, not on a demo machine, not in a controlled benchmark but on real datasets that will embarrass you if your algorithm doesn't scale. And the first real lesson of that career, the one that gets drilled into you early and stays, is this: efficiency is not optional. It is the discipline.

We called it Big-O. Still do.

It's a way of asking one question before you write a single line: how does this behave as the problem gets bigger? An O(n) solution scales cleanly, twice the data, twice the work. An O(n²) solution buckles, twice the data, four times the work. The notation matters less than the reflex. You learn to ask the cost question first, because you've watched what happens when you don't, systems that run fine on a hundred inputs and quietly collapse on a hundred million.

So when the prevailing wisdom in the most advanced corner of computing becomes spend more, more reasoning steps, more tokens, more inference compute, more data centers humming somewhere in a desert.. I felt something I couldn't immediately name. Not disagreement exactly. More like the feeling you get when someone casually breaks a rule you didn't realize you'd internalized until just now.

A true finding. A very large extrapolation.

The "spend more" instinct isn't baseless. There's genuine science underneath it.

It's called test-time compute. Let a model think longer before answering, work through more steps, check its own output and it performs better on genuinely hard problems. The benchmarks are real. On complex multi-step reasoning, spending more tokens at inference time yields meaningfully better answers.

That finding is solid. What happened next is where it gets interesting.

The industry took a result that was true for hard reasoning tasks and quietly applied it to everything. Frontier models, the large, expensive, slow ones, became the default regardless of whether the task needed them. Benchmark culture, which has always measured capability and almost never cost, kept pointing upward. More parameters, more context, more steps. And somewhere in that process, efficiency stopped being a virtue and became a sign that you weren't serious.

A true finding became a culture. And culture, unlike findings, doesn't come with error bars.

It was never really a complexity problem. It's a coefficient problem.

Let me be honest about the Big-O comparison, because it's easy to overstate and a careful reader will catch it.

The wasteful version and the lean version of an AI task usually have the same complexity class. Classify twice as many documents and you use roughly twice the tokens, either way. Both are O(n). Neither is secretly O(n²). The lines don't curve.

What differs is the constant factor, the coefficient sitting in front of the n. One approach costs roughly 360 tokens per request. The other costs 96. Same growth rate. Wildly different slope.

In a theory class, you're taught to ignore constant factors; Big-O deliberately throws them away. But anyone who's run code in production knows the dirty secret: at scale, the constant factor is the budget. A 3.7x coefficient, multiplied across a billion requests a day, is the difference between one data center and four. The reflex Big-O trains, interrogate the cost before you commit is exactly right. We've just been applying it to the wrong part of the equation and ignoring the part that actually dominates the bill.

What that culture looks like in practice

Here's something concrete. I ran it myself.

Take a simple text classification task, the kind any team building an AI pipeline runs dozens of times a day. One piece of text. One category. That's the whole job.

I ran it two ways. The sloppy way: a frontier model with a vague prompt, analyze and classify this and tell me why. The optimized way: a smaller model with a constrained prompt, one word, no explanation.

Same input. Here's what came back.

The sloppy version returned a beautiful little report. Headers. Numbered sections. A sub-category analysis. A breakdown of the likely audience. All of it correct. None of it asked for. 360 tokens. 7.9 seconds.

The optimized version returned one word.

clinical

96 tokens. 0.6 seconds. The same answer the report had buried inside it.

Let me be precise about what makes one sloppy, because it isn't that the frontier model is bad. It did exactly what it was built to do. I asked it to analyze and explain, so it analyzed and explained, thoroughly, well. The sloppiness wasn't the model's behavior. It was the decision to use it. A model built for complex reasoning, pointed at a task that required none, invited through a lazy prompt to do what it does best. Of course it walked through that door.

That difference feels trivial on a single call. Follow it downstream.

Where the wasted tokens actually go

It's tempting to treat all of this as an accounting problem, tokens are units on an invoice and a bigger invoice is a line item someone in finance worries about. But tokens aren't abstract. They're physical. And following the chain is worth doing slowly, because the end of it is somewhere real.

Every token a model generates is a burst of floating-point operations across a chip. Those operations draw current. The current becomes heat. The heat has to be removed, which means cooling air handlers, chilled water, sometimes millions of cubic meters of it a year. None of this lives in the cloud. It lives in a building, on a grid, in a town, drawing power that has to come from somewhere.

The part that matters for this argument is which half of the workload is growing. It isn't training. Some estimates put as much as 80 to 90% of AI computing power on inference, the day-to-day running of models, not their creation. Training a model is a one-time spike. Inference is the permanent load. It's every classification call, every summarization, every query that didn't need a frontier model but got one anyway, multiplied across billions of requests a day, forever.

That's where the wedge in Figure 2 actually ends up. The gap between 360 tokens and 96 isn't a rounding error on a bill. The reflex to spend tokens freely was formed in a context where the cost looked invisible, a number ticking up in an API dashboard. It is not invisible. It has an address. That is not thermodynamics as metaphor. That is just thermodynamics.

The gap nobody has closed

Here's the part that should bother us more than it does.

There is no universally agreed, quantitative framework for choosing which model to use for a given problem. None. We have subjective heuristics, use the small one for easy stuff and the big one for hard stuff and a scattered research literature full of competing routing schemes that each optimize for something different: one for cost, one for latency, one for accuracy, one for human preference. What we don't have is a shared, rigorous answer to the most basic engineering question in the field: given this task, what is the least compute that solves it well?

Think about how strange that is. We can state the computational complexity of a sorting algorithm to the symbol. But ask whether a classification task needs a frontier model or a small one and the honest answer across the entire industry is a shrug and a vibe.

This is the gap. Not that efficient models don't exist; they do. The gap is that every major AI lab can tell you how capable their model is, on a dozen leaderboards, to two decimal places and not one can hand you a quantitative model for when you should reach for it and when you shouldn't. We've built extraordinary engines and no fuel gauge.

The question the model won't ask

I want to be careful, because the point isn't that powerful models are wasteful. On genuinely hard problems, novel hypothesis generation, multi-step reasoning, tasks where the answer space is truly open, spending the compute is right. The capability is real and it matters.

The point is that we stopped asking whether the problem was actually hard.

The companies building these models have every incentive to frame more compute as more progress and that's not a conspiracy; it's just how the economics line up. But the culture that absorbed that framing and turned it into an engineering default made a quieter mistake. It stopped applying the oldest question in the discipline.

Does this need to be this complicated?

Not: does it work. Not: is it the most capable option available. Just: does this problem actually require this much.

That question is a reflex for anyone trained in computational complexity. You ask it before you write a line. The model on the other end of your API call doesn't have that reflex. It will use what you give it. It will think as long as you let it. It will burn exactly as many tokens as the prompt invites.