Dataloader Performance Across 6 GPU Latency Steps

Epoch Avg (ms) — lower is better

Cells highlighted in green are best for that latency.

GPU Utilization (%) — higher is better

Cells highlighted in green are best for that latency.

Wait P95 (ms) — lower is better

Cells highlighted in green are best for that latency.

Epoch Avg (ms) — lower is better

Cells highlighted in green are best for that latency.

GPU Utilization (%) — higher is better

Cells highlighted in green are best for that latency.

Wait P95 (ms) — lower is better

Cells highlighted in green are best for that latency.

Epoch Avg (ms) — lower is better

Cells highlighted in green are best for that latency.

GPU Utilization (%) — higher is better

Cells highlighted in green are best for that latency.

Wait P95 (ms) — lower is better

Cells highlighted in green are best for that latency.

GPU Starvation Is the Default State for PyTorch Loaders

The starkest result in Test 1 is the PyTorch/cached loader. Despite using a per-worker LRU cache to eliminate redundant video decoding, it achieves only 8 to 44% estimated GPU utilization across the six latency targets. Wait P95 values range from 2,400 ms to over 5,800 ms, meaning at the 95th percentile the GPU is sitting idle for nearly six seconds waiting for a single batch. This is not a configuration problem. It reflects the fundamental overhead of multiprocess worker coordination, memory sharing, and collation in PyTorch DataLoader when video data is involved. Even at 300 ms GPU step time, the epoch average barely improves compared to 50 ms: the loader, not the compute, is the bottleneck.

Knonik Runs the GPU at Near-Saturation Across All Latencies

All Knonik loader variants (Batched, Pipelined, and OnDemand) maintain wait P95 values between 0.59 ms and 0.95 ms across every tested latency. GPU utilization is consistently 94 to 99%. The data pipeline essentially disappears from the training loop: the GPU is never waiting for data in any measurable sense. The difference is most pronounced at low GPU latencies (50 to 100 ms), where data pipeline efficiency matters most. At 50 ms, Knonik achieves epoch times of 7,000 to 8,000 ms while PyTorch/naive takes 17,000 ms and PyTorch/cached takes over 110,000 ms.

mmap Disk Caching Offers Marginal Gains in the Baseline Test

Knonik's mmap variants (Batched+mmap, Pipelined+mmap) show small performance differences versus their in-memory counterparts, sometimes slightly faster, sometimes slightly slower depending on latency. The baseline dataset fits well within the OS page cache, meaning mmap's benefit of avoiding repeated decode work is already captured by the kernel's file cache. The mmap variants become more meaningful on larger datasets where decoded tensors exceed available RAM.

HDF5 Is a Significant Bottleneck Compared to Knonik (Test 2)

In Test 2, the HDF5/PyTorch loader reads uncompressed HDF5 files, the most common format in robot learning codebases. Despite HDF5 being an established format with chunked I/O support, it introduces substantial wait latency compared to Knonik loading the same data in Knonik-compressed format. This cost compounds across epochs: because HDF5 files are shared across workers via memory-mapped file handles, the loader struggles to hide I/O latency even with prefetching. Knonik's compressed format is designed specifically for chunk-sequential random access across episodes, which maps efficiently onto the access pattern of trajectory training.

The Efficiency Gap Is Largest at Low GPU Step Times

Across both tests, the performance gap between Knonik and alternatives narrows as GPU step time increases. At 300 ms, even the PyTorch/naive loader can keep up reasonably well because the GPU is slow enough that any pipeline has time to prefetch the next batch. As model training gets faster through hardware improvements and efficient architectures, the dataloader becomes an increasingly critical bottleneck.