What did two years of speed leave behind?
The short version
The buildout had one KPI, time to capacity, and it hit that KPI. Optimization was never on the scoreboard.
- The five largest hyperscalers will spend roughly $750B on capex in 2026
- Frontier training clusters run near full while measured enterprise GPU fleets average about 5% utilization
- Most inference still runs uncached, or cached only inside the provider's own data center
For two years the AI buildout rewarded exactly one skill: getting capacity live before a rival did. The five largest hyperscalers are on pace to spend roughly $750 billion on capital expenditure in 2026 alone. Nobody in that race was promoted for raising GPU utilization, cutting tokens per task, or caching a repeated prompt. People were promoted for energizing buildings. The result is a stack that is extraordinarily fast at adding supply and remarkably casual about using it.
The waste is real, but it is not evenly spread, and honesty about where it sits matters for everything that follows. There is no single utilization number for AI. There are at least two markets wearing one label.
| Corner of the market | What the evidence shows |
|---|---|
| Dedicated frontier LLM clusters | Median utilization of 97% to 99% in published cluster studies (USENIX NSDI 2024) |
| Deployed AI infrastructure overall | Near 100% utilization, per Bank of America (October 16, 2026), with rental rates on A100, H100 and B200 generations up 12% to 37% in 2026 |
| Enterprise GPU fleets on Kubernetes | About 5% average utilization, per Cast AI's 2026 report across tens of thousands of production clusters |
| Operators self-reporting | 86% of 573 surveyed technical leaders report utilization of 50% or less (VentureBeat Research) |
| Independent single-tenant data centers | Under 15% average utilization, per the National Compute Network consortium's own document, not independently audited |
Read together, the table says the shortage and the slack coexist. Frontier training time and rented frontier inference are genuinely scarce, and priced like it. The average enterprise fleet, bought in the same panic, idles. The same split shows up in serving: an agent workflow may make dozens of sequential model calls, resending largely the same context each time, and in most deployments every one of those calls is computed from scratch wherever the capacity happens to sit. The optimization layer, the discipline of removing work instead of adding machines, is the one layer the speed era never built.
Why does the efficiency wave arrive now?
The short version
Three walls bind at once, and none of them responds to money quickly. That is what makes this wave different from a budget cycle.
- Capital: buyers have started auditing rented capacity line by line
- Power: grid connection takes about five years against build cycles of one to two
- Memory: 2026 high bandwidth memory is sold out, and relief is dated 2028
Capital first. The late October repricing of the capex trade, which we tracked in The Capex Crack, Week Two, was the market's opening bid. The operating version arrived on November 6, when Bloomberg reported that Microsoft is winding down its 2023 arrangement to rent AI computing capacity from Oracle and reassessing more than $80 billion of outside capacity contracts with providers including CoreWeave, Nscale and Nebius. Oracle shares fell about 4% on the report. Both sides deserve stating: the reason given is that the rented servers run an older generation of NVIDIA chips that no longer meets Microsoft's specifications, and Oracle says demand for its infrastructure, including earlier-generation GPUs, remains strong, pointing to four-year-old GPU contracts renewed in September at 20% premiums. Even the friendliest reading is a sophisticated buyer auditing rented capacity contract by contract. That audit habit, once formed, does not stay confined to one vendor.
Power second. Lawrence Berkeley National Laboratory data puts the median journey from a grid interconnection request to commercial operation at roughly five years, with waits of four to seven years in the busiest markets, while a data center itself takes one to two years to build. Transformer lead times run near four years on PwC's analysis. A hyperscaler can write a check for GPUs in a quarter; it cannot write a check that moves its place in the interconnection queue. When energised capacity is the fixed quantity, output per megawatt becomes the growth variable, and output per megawatt is an efficiency metric.
Memory third. As we laid out in The Memory Play, 2026 high bandwidth memory supply is sold out across SK hynix, Samsung and Micron, conventional DRAM prices rose about fivefold in a year as wafer capacity shifted into HBM, and the next real supply relief is dated 2028. Memory is not just an input cost here. Generating every token streams model weights and cached context through memory, so techniques that cut memory per token, reusing cached context instead of recomputing it, quantizing caches, shrinking attention state, act arithmetically like new memory supply. When the scarcest input cannot be bought, economizing it is the only supply increase on offer.
The shared property. Capital can be repriced overnight, but it cannot print grid connections or HBM wafers. All three walls push in the same direction: more output from the capacity, power and memory that already exist. That is the definition of an efficiency wave.
What is NVIDIA selling when it sells efficiency?
The short version
The silicon vendor saw the wave first, and it sells the shovels for this phase too: serving software that raises tokens per GPU on hardware already shipped.
- Dynamo is NVIDIA's open-source inference operating system, with a cache-aware router at its center
- NVIDIA claims up to 30x more tokens per GPU on a flagship reasoning model, a vendor figure
- Independent MLPerf results show software alone lifting throughput up to 1.6x on unchanged hardware
NVIDIA Dynamo, the open-source inference software NVIDIA introduced as the successor to its Triton serving stack, is built around exactly the work this article is about: splitting prefill and decode across different machines, planning GPU allocation dynamically, and routing each request with knowledge of where its cached context already lives. A production 1.0 release followed in March 2026. NVIDIA's own launch claims are large: it says Dynamo doubles the performance and revenue of factories serving Llama models on the existing Hopper generation, and raises tokens generated per GPU by more than 30x for the DeepSeek-R1 reasoning model on GB200 NVL72 racks. Those are vendor numbers from NVIDIA's newsroom, presented here as claims, not measurements.
The independent evidence that software is a first-order lever, not a marketing one, comes from MLCommons. In the MLPerf Inference v6.1 results published in September 2026, software optimizations alone improved throughput on the same GB300 hardware by up to 1.6x over the prior round, and NVIDIA's Vera Rubin preview systems posted up to 3.7x the throughput of GB300 on one benchmark suite. A 60% gain from software, on silicon already bought and powered, is the efficiency wave in miniature: no new building, no new grid connection, no new wafers.
There are two readings of NVIDIA's position, and the article does not need to pick one. In the first, efficiency strengthens the incumbent: tokens per watt is the product, NVIDIA sells it at both the silicon and software layers, and every efficiency gain raises the value of its installed base. In the second, Dynamo being open source and framework-agnostic, running on vLLM, SGLang and TensorRT-LLM, means NVIDIA is deliberately commoditizing the serving layer to protect the silicon layer beneath it. Both readings agree on the fact that matters here: the company with the most to lose from an efficiency wave moved earliest to own it.
What are the frontier labs doing inside the model?
The short version
The labs are capturing a large share of the efficiency gain inside their own walls, through prompt caching, smaller models and collapsing token prices. This is the strongest version of the counter-thesis.
- Cached input tokens are discounted up to 90% at Anthropic and Google, and up to 98% at DeepSeek
- Small models now take most agent turns at a tenth of flagship prices
- Per-token prices have fallen about 1000x in three years while total enterprise AI spend tripled to $37B in 2025
Prompt caching is the model layer's answer to repeated work. An agent resends its system prompt, tool definitions and conversation history on every turn; caching that prefix means the provider computes it once and charges a fraction for every reuse. The discounts now on published price lists are not marginal.
| Provider | Mechanism | Discount on cached input |
|---|---|---|
| Anthropic | Explicit cache breakpoints set by the developer, five-minute default lifetime | 90%, with cache reads priced at a tenth of standard input on its Sonnet tier |
| OpenAI | Automatic prefix caching; newest model families add paid cache writes at longer lifetimes | 50% at introduction, rising to as much as 90% on the newest models |
| Gemini context caching, implicit and explicit | 90% on reads, with hourly storage charges for held context | |
| DeepSeek | Automatic disk-backed prefix caching from the first token | Up to 98%, a vendor figure |
The second lever is model size. Anthropic's Haiku 5.5, released on October 7, 2026, is priced at $0.10 per million input tokens for most prompts, and Anthropic describes it as roughly 75% cheaper to run than the model it replaces; OpenAI cut prices on its own small and mid-tier models by about half in the same period. Production agent systems increasingly route the bulk of turns to these small models and escalate only the hard steps. The third lever is openness: in October 2026 DeepSeek published DeepInfer, the serving framework behind its own production API, putting frontier-grade serving optimizations into public code.
The backdrop is a price collapse that has not reduced spending. Menlo Ventures estimates enterprise generative AI spending tripled to $37 billion in 2025, over a period when per-token prices fell roughly 1000x. Cheaper tokens have reliably meant more tokens. That is the Jevons pattern, and it matters for the bear cases later in this piece.
Now the counter-thesis, stated at full strength, because the rest of this article depends on taking it seriously. Everything in this section happens inside the lab's own data centers, on the lab's own hardware, under the lab's own pricing. Prompt caching reduces the lab's serving cost and the customer's bill without any network in between. If the efficiency frontier keeps being captured at the model layer, the infrastructure between the user and the model commoditizes into a pipe, and the toll stays with Anthropic, OpenAI and Google. Anyone underwriting the network layer's version of this wave is betting against that paragraph.
What can the network layer actually do?
The short version
Two companies operate global edge networks that could cache and route AI work near the user. One has disclosed semantic caching without metrics. The other has shipped the simpler exact-match version and labels the semantic one planned.
- Akamai routes across edge, regional and core tiers with an orchestrator built for cost per token
- Cloudflare's AI Gateway caching is exact-match today; its docs describe semantic caching as planned
- Neither network has published cache hit rates on real agentic traffic, the metric that would settle the claim
The network layer's physical claim is different from the labs'. A model-layer cache helps when the same provider sees the same prefix again. A network cache sits in front of every provider, near the user, and an agent task is not one call but dozens of sequential ones: shaving cost and latency at every step compounds in a way a single data center round trip cannot. Routing is the other half: sending each call to the cheapest tier that meets its latency target, and away from capacity that is idle in the wrong place or scarce in the right one. This is the discipline content delivery networks spent 25 years building for web objects, pointed at inference.
| Capability | Akamai | Cloudflare |
|---|---|---|
| Network | More than 4400 locations, many embedded inside access networks; inference GPUs in about 20 regions at the October 2025 Inference Cloud launch | Network in more than 300 cities; Workers AI GPUs in nearly 200 cities as of its 2025 developer announcements |
| Caching for AI responses | Semantic caching disclosed as a capability of its AI Grid orchestrator; no hit rates or savings metrics published | AI Gateway response caching shipped, exact-match on a hash of the full request; semantic caching described as planned in Cloudflare's own documentation |
| Prompt and prefix caching | Cache-aware routing inside the orchestrator; implementation detail undisclosed | Workers AI prefix caching shipped in 2025, alongside speculative decoding speedups |
| Commercial proof points | More than $14B of multi-year cloud infrastructure commitments signed in 2026, including Anthropic's $11.6B; CIS revenue of $109M in Q3 2026, up 34% | Q3 2026 revenue of $782.5M, up 39%, its fastest growth in nearly four years; first revenue from AI bot traffic; an $11M pool-of-funds Workers contract with a fast-growing AI customer |
Akamai's version is the more ambitious architecture and the less proven one. Its orchestrator, built as the first global implementation of NVIDIA's AI Grid reference design (our deep dive is here), routes each request across edge, regional and core tiers by cost per token, time to first token and throughput. The commercial ramp is now dated: in its Q3 2026 report, Akamai guided 2027 revenue growth of 17% to 22% in constant currency, cloud infrastructure revenue of $1.1B to $1.3B in 2027 against a $109M quarter today, and 2028 revenue above $7.0B, while spending through a capex peak that includes $2.445B to $2.645B guided for Q4 2026 alone. Management frames the large deals at about a 13% after-tax IRR floor and roughly 75% adjusted EBITDA margins at full ramp. Those are management's figures, offered on the earnings call, and they describe a plan to be the efficiency layer for other companies' AI, CPUs near users instead of GPU factories far away.
Cloudflare's version is narrower today and further along commercially. Its AI Gateway gives any customer a caching, logging and routing layer in front of the major model providers, free at the core tier; the caching that has actually shipped is exact-match, which helps repeated identical requests and does little for personalized agent traffic. Its own Q3 2026 print, reported November 5, shows both the opportunity and the cost of the serving business: revenue up 39% to $782.5M with Workers and agentic workloads cited as drivers, while GAAP gross margin fell about three points to 71.2% and free cash flow margin slipped to 10% from 13%, as the cost of serving AI traffic grew faster than sales. Even the company best positioned as the toll collector is feeling the efficiency squeeze inside its own income statement.
What the market currently pays for the two networks frames the disagreement without settling it. Prices are as of the November 6, 2026 close, and revenue figures are each company's own 2026 guidance.
| November 6, 2026 | Akamai | Cloudflare |
|---|---|---|
| Share price | $83.09 | $268.75 |
| Market capitalization | $11.94B | About $95B to $97B |
| 2026 revenue guidance | $4.44B to $4.465B | $2.963B to $2.964B |
| Market cap per dollar of guided 2026 revenue | About 2.7x | About 32x |
One honest gap applies to both networks. Neither has published the number that would prove its layer matters: cache hit rates, and savings attributable to them, on real agentic traffic at production scale. Akamai disclosed the capability and no metrics. Cloudflare shipped the simpler mechanism and labeled the ambitious one planned. Until those numbers exist, the network layer's claim on the efficiency wave is an architecture and a contract book, not a measurement. Our earlier head-to-head, Akamai vs Cloudflare: Two Edges, One Scorecard, covers the two businesses in full; this article takes no side in it.
Who wins and who pays if the wave lands?
The short version
Efficiency is not a demand bear case; history says it enlarges the market. The pain concentrates in business models that sell brute-force capacity by the hour or the lease.
- Token prices fell about 1000x and total spend tripled: the Jevons pattern is the base case
- Landlord and neocloud capacity models carry the repricing risk, on different clocks
- Microsoft's Oracle non-renewal is the template event, and Oracle's premium renewals are the live counter-evidence
Start with the side most often gotten wrong. Efficiency waves in computing have historically grown the market they optimize. Cloud cost optimization did not shrink cloud spending; it funded more workloads. The token data in this article says the same: prices down roughly 1000x, enterprise spend tripled. A world where inference gets 40% cheaper is, on the record so far, a world that runs more inference, which is why NVIDIA can sell the efficiency software and the silicon in the same motion.
The toll, though, can only be collected once per unit of work, and three layers are reaching for it. Silicon collects when efficiency ships as a feature of the next GPU and its software. The model layer collects when caching and routing stay inside the provider's API, invisible and unbypassable. The network collects when traffic is multi-provider, latency-sensitive and repetitive enough that a nearby cache and a smart route beat a distant data center. Real traffic is all three kinds at once, which is why the likely outcome is a split toll rather than a single winner, with the proportions set by execution over the next two years.
Who pays is clearer. The deflationary edge of the wave lands on models that sell undifferentiated capacity: build a building, fill it with GPUs, rent the hours. Developer-landlord structures on 15-year leases defer the reckoning the longest and concentrate it in tenant credit and end-of-lease residual values. Neocloud service contracts with hyperscalers reprice sooner, at renewal, which is why the Microsoft and Oracle story matters beyond its size: it is the first visible renewal negotiation of the efficiency era, and it ended with the buyer walking from capacity it judged obsolete. The counter-evidence is equally visible and equally current: Oracle renewing four-year-old GPU contracts at 20% premiums in September says that in a genuinely scarce market, even old capacity reprices upward. Both facts are true at once. That is what a transition looks like from inside it.
What to watch, with dates
Akamai: the Q4 2026 capex print in its February 2027 report; first Anthropic services and first payment, scheduled for late Q2 2027, which starts the warrant clock and the revenue clock together; 2027 cloud infrastructure revenue against the $1.1B to $1.3B guide. Cloudflare: whether AI-attached revenue gets its own disclosure line, and whether gross margin stabilizes as serving scales. NVIDIA: Dynamo adoption in third-party benchmarks rather than its own. The labs: whether cached-token discounts keep widening, which would say the model layer is winning the toll outright.
What would prove the wave is not arriving?
A landscape piece owes its reader the falsifiers. Four developments, any two together, would say the speed era is intact and this article's premise is early or wrong.
Frontier scarcity persists untouched. If deployed utilization stays near Bank of America's near-100% reading and GPU rental rates keep climbing through 2027, capacity remains the scarce good and efficiency remains a footnote in enterprise budgets rather than a strategy.
Capital discipline proves to be a mood. Street estimates already put 2027 hyperscaler capex above $1 trillion. If that spending lands with no spread of the Microsoft-style audit habit, no further non-renewals and no returns framework demanded by investors, the capital wall described above was weather, not climate.
Memory loosens early. Our Memory Play dates real HBM relief to 2028, with high bandwidth flash and faster supply as the named risks to that date. A 2027 loosening would remove the wall that makes memory-saving techniques worth a premium.
The metrics never appear. If by the end of 2027 neither edge network has published cache hit rates or efficiency-attributed revenue, and the labs' internal caching keeps absorbing the gains, the network layer's claim was a story told over an architecture diagram. Inefficiency is also a stable equilibrium: enterprise fleets have idled at low utilization for years because no one's compensation depended on fixing it. Waves need a forcing function. This article's claim is that capital, power and memory have finally supplied three at once. The falsifiers above are how a reader can grade that claim without taking our word for it.
The land in one line. The buildout spent two years buying speed at any price; the next phase pays whoever removes work from the system, and the only genuinely open question is which layer of the stack, silicon, model or network, collects for removing it.
Frequently asked questions
Why is AI efficiency becoming urgent now?
Three constraints bind at once. Capital is repricing after the late October capex selloff; grid connections take about five years against build cycles of one to two; and 2026 high bandwidth memory is sold out. When none of the inputs can be bought quickly, getting more from installed capacity is the only lever left.
Which layer captures the efficiency value: chips, models, or networks?
All three are collecting today on different traffic. NVIDIA sells efficiency in its serving software, the labs discount cached tokens by up to 90 percent inside their own walls, and the two global edge networks are building tiered caching and routing. The split a few years out is the open question this article maps.
What would prove the efficiency wave is not coming?
Watch four things. Frontier utilization and GPU rents staying at ceiling levels through 2027; hyperscaler capex re-accelerating with no audits of rented capacity; memory supply loosening a year early; and neither edge network ever publishing cache metrics. Any two of those together would mean speed is still the only KPI.
Disclosures
Not financial advice. This article is published for educational purposes only. Nothing here is a recommendation to buy, sell, or hold any security, and nothing here is financial advice. Investing involves risk, including the possible loss of principal.
Positions. The author holds long positions in Akamai and Nebius, both discussed in this article. The author holds no position in Cloudflare, NVIDIA, Applied Digital, Microsoft, Oracle or CoreWeave.
Accuracy. Utilization figures are from the sources named beside them: USENIX NSDI 2024 cluster research, Bank of America commentary of October 16, 2026, Cast AI's 2026 State of Kubernetes Optimization report, a VentureBeat Research survey of 573 technical leaders, and the National Compute Network consortium's own document, which is labeled in the text as unaudited. Hyperscaler capex totals are Rabobank's compilation for the five largest spenders. Grid interconnection timelines are Lawrence Berkeley National Laboratory data as compiled in industry reporting, and transformer lead times are PwC analysis. The Microsoft and Oracle account is Bloomberg reporting of November 6, 2026 as carried by Reuters, with Oracle's response from the same reporting; neither company disclosed the contract's size. NVIDIA's Dynamo performance figures are NVIDIA's own claims from its newsroom releases and are labeled as claims in the text; MLPerf figures are MLCommons MLPerf Inference v6.1 published results. Prompt caching discounts are from the providers' published pricing and documentation as of early November 2026; DeepSeek's figure is its own. Enterprise AI spending and token price history are Menlo Ventures estimates. Cloudflare's caching status is from Cloudflare's developer documentation as compared in October 2026; its financials are from its Q3 2026 release. Akamai's financials, guidance and deal return figures are from its Q3 2026 release and earnings call, and the return figures are management's framing, labeled as such. Share prices and market capitalizations are as of the November 6, 2026 close. Errors can occur and facts change; verify independently before acting on anything you read.
No advisory relationship. Reading this site does not create an advisor-client relationship of any kind. See the full disclosures page.