Google claims its new AI software cuts inference latency by over 90% — but there’s a catch

The article looks at TPU Sync, a new open-source system from Google meant to reduce the time AI models spend moving data between accelerator memory, host RAM, storage, and the network. It is aimed at developers running large models with very long context windows, where that data movement becomes a bottleneck. Google reports that in some benchmarks it cut KV-cache transfer times and time to first token by as much as 90 percent, though results depend heavily on workload, how much cached data can be reused, and available bandwidth. The author notes that analysts see Nvidia systems still ahead in parts of the disaggregated-serving performance picture, and no fully tuned head-to-head comparison has been published yet. The software works with popular inference engines such as vLLM and SGLang, which could let teams try TPUs without rebuilding their stack. It is not yet production-ready: the APIs are still changing, and support is limited to specific Cloud TPU versions and framework releases.