Open Source Summit + AI_dev Japan 2024

As GPUs become increasingly powerful, the separation between compute and storage often results in underutilized GPUs waiting for data. Meanwhile, high-performance components on GPU machines, such as NVMe storage and fast networks leveraging InfiniBand or special NICs, remain idle. Effectively leveraging these hardware resources to address GPU underutilization is a critical challenge. In this talk, we introduce a Kubernetes-native distributed caching layer that leverages NVMe disks and fast networks to optimize PyTorch training data access. Utilizing stateless workers for scalability and ETCD for membership services, this caching layer efficiently manages and serves data. Cached data is rapidly and efficiently fed into GPU memory using NVIDIA's DALI data loader, GPUDirect Storage (GDS), and Remote Direct Memory Access (RDMA), significantly reducing data transfer bottlenecks and improving overall training performance.