In his Interconnects newsletter, AI researcher Nathan Lambert wrote that many aspects of the AI training and inference stack are highly verifiable. Training metrics, such as tokens per GPU per second, essentially represent training speed. Inference metrics, such as tokens per prompt and FLOPs per token, ultimately reflect the cost per answer. Both are highly optimizable, especially with mature sub-problems and architectural tradeoffs to leverage. I anticipate that within a few years, AI agents will help optimize the entire process end-to-end, bringing our inference capabilities close to the underlying computing power limits of accelerators like GPUs. In recent years, companies have already seen significant gains in inference efficiency—for example, after releasing a model at a certain price, they can save an additional 10–30% in service costs. The entire stack will continue to compound, and I expect the effective cost of model intelligence to decrease almost exponentially in the coming years (possibly faster than recent trends).
These low-hanging fruits of efficiency can be harvested within a few years. In the longer term, the co-design of accelerators and models has an even longer timeline, potentially bringing several orders of magnitude more efficiency improvements on flexible GPU platforms. During this period of rapid efficiency improvements, the flexibility of GPUs will be beneficial because the architectural exploration space is crucial and a major lever for automated research. In my view, it is reasonable to predict that pre-training research (at least in terms of architecture and data selection for current models) will be automated within 2–3 years.
