Which AI bottleneck takes longest to fix?
Power takes years to expand. Models can change in months. Our read: secure slow-moving dependencies early, and spend against the bottleneck you have measured.
· 6 min read
Signalful Analysis
From interviews with Walter Goodwin of Fractile, Thomas Sohmers of Positron AI, Stefano Ermon of Inception, Amin Vahdat of Google, Zach Dell of Base Power, Jensen Huang of Nvidia and Pat Gelsinger of Playground Global
Google is expected to spend more than $200 billion on AI infrastructure this year, mostly on data centers. Yet a voice-agent customer cut costs by switching models on ordinary Nvidia GPUs. Both are responses to constraints, but they require different commitments. Which AI bottleneck takes longest to fix, and what should you do about it?
Builders give four answers: memory, power, the algorithm or the build-out of sites and crews. Power looks hardest to expand. But the industry's slowest fix is not necessarily your company's immediate constraint. Your priority is what stops you meeting demand, delivering acceptable performance or earning a margin.
Our reading: secure slow-moving dependencies early and keep fast-changing technology choices flexible. Power is the most persistent physical constraint in these accounts; spend against the bottleneck you have measured in your own business.
Four answers
The first answer is memory. Walter Goodwin, founder and CEO of Fractile, says inference economics at data center scale come down to memory cost per gigabyte. Thomas Sohmers, co-founder and CTO of Positron AI, says Nvidia GPU compute grew about 120 times between 2014 and 2024, while memory bandwidth improved only 17 times.
Stefano Ermon, a Stanford professor and co-founder and CEO of Inception, says writing one token at a time leaves GPUs moving weights through memory. More compute alone does not address the problem Ermon describes. The question for an operator is whether a different model or memory architecture improves the actual workload.
The second answer is power. Amin Vahdat, who runs AI infrastructure at Google, calls it the most fundamental constraint. Zach Dell, co-founder and CEO of Base Power, expects US electricity demand growth to move from about 2% a year toward 10%. Dell says the grid is too small.
Quick fixes have limits. Batteries move electricity from midday to evening; they add no generation, Dell says. Vahdat says supplying Google's needs independently with solar and batteries at 99.99% reliability would mean building about 2 gigawatts to get 1. Capacity and dependable capacity are different requirements.
The third answer is the algorithm. Ermon says diffusion writes many tokens at once, making better use of parallel hardware. A voice-agent customer moved from models served on Cerebras chips to Inception's on ordinary Nvidia GPUs and got the same speed at lower cost. That is a reason to test model changes before committing to specialized infrastructure, not proof that the same switch suits every workload.
The fourth answer is the build-out. Jensen Huang, founder and CEO of Nvidia, points to land permitting, generation, skilled construction workers and local moratoriums. Huang expects order books to exceed supply for a few years. A chip order does not resolve the dependencies between buying equipment and putting it to work.
Cost, speed, place, time
Four tests tell you which bottleneck is yours: what each task costs, how fast it runs, where the hardware goes and when it has to be live.
Spotted an error? Email corrections@signalful.com and we’ll fix it.
