1.How are AI workloads exposing hidden inefficiencies in traditional data center infrastructure?
The massive parallelism in the GPUs processing AI workloads is perhaps the root cause of exposing the limitations of the traditional compute-storage-networking infrastructure. GPUs, with their thousands of processor cores working in parallel, stress the infrastructure differently than traditional CPUs with a few dozen cores each. The GPU clusters running AI workloads want to churn through larger datasets more rapidly than x86 processors running traditional enterprise applications do. This means orders of magnitude higher demands for IOPs (Input-Output operations per second) with smaller chunks of data per IO (Input/Output). Also, x86 processors employ deep instruction pipelines, making them less sensitive to small variances in data access latency. GPUs run shallow instruction queues, with AI workloads often executing the same instruction on many data elements, making the workload highly sensitive to delays in getting that next piece of data.
These differences in handling and accessing data expose bottlenecks in the network connections between the processing nodes and the storage nodes, in the capacity and bandwidth of local memory, and in the data transfers from local drives to system memory and GPU caches. All of these bottlenecks impair the utilization rates and overall efficiency of the AI infrastructure.
2.In what ways does smarter data management improve performance while reducing environmental impact?
There are two central aspects of data management that contribute to improving application performance and overall infrastructure efficiency. First is the software aspect.
In software, intelligent placement of data across tiers is crucial to improving the output per processor and to reducing the energy consumed in moving data – both of which impact efficiency a.k.a. “work per watt”. For example, hyperscale data center operators carefully study their data access patterns to determine the data’s “hotness” (how actively a given data element is being accessed by the processors and how likely that element is to be accessed again in the near future). Based on the data’s hotness, the data is kept in local cache or moved out to DRAM, local SSDs, a high-capacity Flash tier, networked HDD, or even Tape storage. The further down the ladder the data is moved, the more expensive that data is to bring back up to the processors (in terms of both time and electrical power).
Second is the hardware aspect. The memory, storage, and networking hardware can all contribute to improving or impairing performance and power efficiency. High-bandwidth memory attached to the GPUs, local SSDs optimized for low-latency checkpointing and random reads, and networked Flash storage solutions designed to ensure that every byte of network bandwidth can be filled, all improve efficiency.
3.Why are traditional SSD architectures struggling to keep up with AI applications, and what role do next-generation storage controllers play in balancing performance and power constraints?
Traditional SSD architectures are built around 4KB and larger IO requests. As long as a read request is 4KB or larger, data center NVMe SSDs are able to fully utilize the PCIe bus bandwidth. However, the SSDs cannot scale IOPs to keep the PCIe bandwidth filled for smaller IOs. For example, PCIe 5 SSDs achieve ~13GB/s of throughput at 3.3M IOPs with 4KB IOs. With 8KB IOs, the IOPs drop in half, but the total bandwidth is maintained. Conversely, when the IO size drops to 2KB, 1KB, or 512B, the SSD cannot scale up beyond the 3.3M IOPs. This means that 50%, 75% or even 87.5% of the PCIe bus bandwidth goes unutilized. Enterprise applications such as databases and analytics primarily use larger IO elements, from 4KB to MBs. AI workloads want the 512B IOs that expose the limitations of current SSDs.
Innovations to both the NAND media and the SSD controllers are required to overcome this challenge.
4.What critical features of modern storage solutions are essential to mitigate the inefficiencies in compute and latency in AI workflows?
Modern storage solutions must be designed to (1) “keep the pipes full” regardless of the IO size or access pattern, (2) enable larger capacities higher up in the memory-storage hierarchy to allow low latency access to massive data sets, (3) ensure that checkpoint writes can be completed quickly so the GPUs can move on to the next task.
New stages in the memory-storage hierarchy have emerged and are emerging to address these design needs. Recently, high-bandwidth memory brought larger memory caches directly into the GPU package. CXL (compute express link) technology has also begun to ramp as a means to bring higher capacities and higher total bandwidth into the memory tier. CXL attaches DRAM via the PCIe bus instead of coupling it with CPUs. CXL will have higher latency than CPU-attached DRAM, but significantly lower latency and finer-grain access than SSDs. New AI-optimized SSD controllers are in development to address the need for smaller IO transfers.
5.What key features of ScaleFlux’s FX5016 controller set it apart from other solutions in the market in meeting the demands of AI and data-heavy workloads?
The ScaleFlux FX5016 controller integrates data compression engines and intelligent metadata management functions, which are not commercially available in other controllers. These two features work in concert to reduce the time and energy consumed in writing data to the flash. This results in lower latency for write operations, higher sustained write throughput, and higher performance in mixed read-write workloads—all of which are essential to rapidly completing checkpoints while delivering data to the GPUs in the AI processes.
- About JB Baker
- About ScaleFlux
JB Baker, the Vice President of Products at ScaleFlux, is a successful technology business leader with a 20+ year track record of drivingtop and bottom-line growth through new products for enterprise and data center storage.After gaining extensive experience in enterprise data storage and Flash technologies with Intel, LSI, and Seagate, he joined ScaleFlux in 2018 to lead Product Planning & Marketing as the company innovates efficiencies for the data pipeline.He earned his BA from Harvard and his MBA from Cornell’s Johnson School. ScaleFlux’s website: https://scaleflux.com/
In an era where data reigns supreme, ScaleFlux emerges as the vanguard of enterprise storage and memory technology, poised to redefine the landscape of the data infrastructure – from cloud to AI, enterprise, and edge computing. With a commitment to innovation, ScaleFlux introduces a revolutionary approach to storage and memory that seamlessly combines hardware and software, designed to unlock unprecedented performance, efficiency, security, and scalability for data-intensive applications. As the world stands on the brink of a data explosion, ScaleFlux’s cutting-edge technology offers a beacon of hope, promising not just to manage the deluge but to transform it into actionable insights and value, heralding a new dawn for businesses and data centers worldwide. For more details, visit ScaleFlux’s website: https://scaleflux.com/
Techedge AI is a niche publication dedicated to keeping its audience at the forefront of the rapidly evolving AI technology landscape. With a sharp focus on emerging trends, groundbreaking innovations, and expert insights, we cover everything from C-suite interviews and industry news to in-depth articles, podcasts, press releases, and guest posts. Join us as we explore the AI technologies shaping tomorrow’s world.
