An article from SK Hynix's Global News Center states that GPUs and AI accelerators are core technologies supporting the training of large models and the processing of complex inference requests. However, in real-world service environments, AI performance depends on more than just computing speed. Even if the processor is ready to execute calculations, expensive accelerators will simply idle and wait if data cannot be delivered in time, dragging down the efficiency of the entire system. Therefore, an AI system can only achieve its full performance when the accelerator's computing power, data delivery speed, and data movement path work in synergy.
The difficulty in achieving this synergy stems from the severe disconnect between the growth of computing power and data delivery capacity.
Over the past 20 years, peak server computing power has increased approximately threefold every two years, but DRAM bandwidth has only increased 1.6 times, and interconnect bandwidth even less than 1.4 times—computing is running incredibly fast, but the speed of "feeding data" is far behind. This ever-expanding "memory wall" is replacing computing power itself as the primary bottleneck for AI systems.
The key to solving this problem lies in designing the system around the data path, rather than focusing solely on computing power. High-frequency data should be stored close to the processor, while massive amounts of data should be efficiently retrieved through storage and networks. It's impossible to place all information in the fastest location; therefore, the storage location and movement path of data must balance speed, capacity, cost, and energy efficiency. Software scheduling is equally crucial: which tasks are processed first, when data is prefetched, and how tasks are allocated—improper scheduling can create new bottlenecks.
This "dataflow-centric" mindset is reshaping the competitive landscape of AI infrastructure.
Fast GPUs, high-bandwidth memory, large-capacity storage, and high-speed networks—no matter how strong any single component is, they cannot support overall performance if they operate independently. Semiconductor manufacturers, server manufacturers, network equipment vendors, cloud service providers, and storage companies can no longer optimize only their own components; they must design collaboratively based on the entire data flow chain.
Faster GPUs remain the core of AI infrastructure, but the question has moved forward: how quickly and efficiently can data reach the GPU and be processed? The next bottleneck determining AI performance begins with this question.
