DataLoader Performance:
4 Experiments, Real Robotics Data

PyTorch/cached decodes a full episode on first access and stores it in a per-worker LRU cache (8 episodes per worker). The theory: pay the decode cost once, reuse the result for all future batches drawn from that episode. In practice, this assumption collapses under shuffled sampling.

With 8 parallel workers, each worker maintains its own independent LRU. There is no shared cache between workers. If worker 0 caches episode 47, worker 1 has no visibility of that - it will decode episode 47 again from scratch the next time it receives a batch request for that episode. Across 8 workers pulling random samples from a large dataset, the probability that the same worker gets the same episode before it falls out of an 8-slot LRU is very low.

The result is near-zero cache hit rate, combined with a higher per-miss cost than naive seeking. PyTorch/naive decodes only chunk_len = 10 frames per __getitem__ call. PyTorch/cached decodes the entire episode on every miss. With a median wait of 1,798 ms and 54.2% of steps classified as slow, the cache is paying the maximum possible cost on every request.

These two metrics measure different things, and the tension between them is real. Epoch average measures total wall-clock time per epoch - it is dominated by the warm epochs once the cross-epoch cache is populated. p95 wait latency measures the 95th-percentile per-batch data fetch time, capturing outlier stalls within a single epoch.

Pipelined's architecture involves parallel decode processes writing into shared memory, with the training loop consuming from that buffer. Most steps receive data with near-zero wait (p50: 1.7 ms). But occasionally the prefetch pipeline stalls - a decode process falls behind, the buffer drains, and the training loop has to wait. That stall shows up as a high p95 (145.5 ms) and a 4.7% slow-step rate.

OnDemand, by contrast, streams data on-the-fly without the overhead of managing a parallel decode pipeline. Its p50 is 18.1 ms and p95 is 94.6 ms - more consistent, but with no cross-epoch cache benefit. By epoch 3, Pipelined is at 1,325 ms/epoch vs OnDemand's 3,804 ms - the cache advantage overwhelms the occasional stall penalty.

OnDemand has no cold-start penalty (346 ms) because it does not pre-decode or pre-fill anything. It fetches and decodes data as each batch is requested. This makes it the fastest loader at epoch 1 by a large margin - 3,886 ms vs 5,824 ms for Pipelined and 12,346 ms for Batched.

However, OnDemand pays the same decoding cost on every epoch because it does not maintain a cross-epoch cache. Each epoch is a fresh pass. Pipelined and Batched both build a RAM cache during their first epoch. From epoch 2 onward, data is served from memory - not decoded from storage. The warm epoch time for Pipelined drops to ~1,325 ms and for Batched to ~1,168 ms. OnDemand stays flat at ~3,804 ms every epoch.

Projected epoch averages for longer runs (using cold-start + warm-epoch amortisation):

The crossover happens around epoch 2–3 for Pipelined vs OnDemand, and around epoch 50 for Batched overtaking Pipelined (Batched's lower warm-epoch time eventually wins over Pipelined's lower cold-start). OnDemand never wins past epoch 1 once caching loaders warm up.

HuggingFace's LeRobot v3 format stores numeric streams (actions, proprioception) in Apache Arrow files - memory-mappable and fast to access. Video frames are stored in compressed formats (h264, webp) and decoded per-chunk via a set_transform() hook at batch time.

In Experiment 3, all loaders are operating on the same VP9 dataset - so the Arrow advantage for numerics doesn't apply; the bottleneck in all cases is video decode. What hurts HuggingFace/Arrow specifically is the set_transform() call overhead: every batch triggers Python-level dispatch into a transform pipeline that decodes, converts, and stacks frames. This adds latency on top of the raw decode cost.

Additionally, HuggingFace's dataset infrastructure is optimised for high-throughput sequential or sliced access patterns - the Arrow columnar layout shines when you read contiguous rows. Random chunk sampling from shuffled episodes produces scattered seeks through the Arrow files, negating the columnar advantage. The result: 15,627 ms epoch average with 19.0% slow steps and a 719 ms p95 - comparable to PyTorch/naive, not better.

GPU utilisation in this benchmark is a synthetic metric derived from the ratio of simulated GPU compute time to total wall-clock step time. Each step simulates a fixed 50 ms matmul workload. GPU utilisation is computed as:

So 69.9% means that for every second of wall-clock training time, approximately 699 ms is the GPU executing the simulated forward/backward pass, and 301 ms is the CPU waiting for the data loader to produce the next batch. It is a measure of the data-loading overhead fraction, not GPU hardware saturation (SM occupancy, memory bandwidth, etc.).

In real training, the GPU compute step is not a fixed 50 ms - it depends on model size, batch size, and hardware. The 50 ms target was chosen to approximate a mid-scale robotics policy (e.g. ACT, Diffusion Policy). For smaller models with faster forward passes, the data-loading fraction becomes even more dominant. For very large models, it matters less. The benchmark is most representative of training workloads where a single step takes 20–100 ms of GPU compute.

wait_avg aggregates all per-batch wait times into a single mean. A loader can have a low wait_avg while still regularly stalling the GPU - if most batches are fast and a few are catastrophically slow, the average stays acceptable but training throughput suffers on every slow batch.

slow_pct measures the fraction of steps where the wait time exceeded some threshold (in this benchmark, a multiple of the median). A 19.0% slow_pct for HuggingFace/Arrow means nearly 1 in 5 batches causes a full GPU stall. During those stalls, the GPU sits completely idle - not partially utilised, but at zero throughput. No amount of fast average batches compensates for the idle time during a stall.

This is why PyTorch/naive (wait_avg: 102 ms, slow_pct: 17.1%) and HuggingFace/Arrow (wait_avg: 121 ms, slow_pct: 19.0%) produce almost identical epoch averages (15,434 ms vs 15,627 ms) despite a 19 ms average wait difference - the epoch runtime is dominated by how many times the GPU fully stalls, not by the average wait across all steps.

Batched pre-decodes all episodes serially before the first epoch begins. The 4,390 ms cold start is the time spent building a complete decoded representation of the dataset in RAM. This is the entire upfront cost paid once, after which every subsequent epoch reads directly from that in-memory structure.

The consequence is that epoch 1 looks expensive (12,346 ms: 4,390 ms cold start + first epoch pass), but epoch 3 drops to 1,168 ms - the lowest warm-epoch time of all six loaders. Batched's p50 wait is just 1.2 ms at warm state, because fetching a batch is essentially a memory copy from a pre-indexed buffer.

Whether this trade-off is worth it depends entirely on how many epochs you are training. At 3 epochs, Batched's average (4,890 ms) is worse than OnDemand (3,815 ms) and Pipelined (2,831 ms). At 50 epochs, Batched becomes the fastest of the three (1,392 ms avg). The cold start amortises completely once you train long enough.

One practical implication: if you are running short experimental loops (1–5 epochs) to validate a model change, Batched is the wrong choice. Switch to OnDemand. If you are running a full training run of 30+ epochs, Batched pays back its cold start many times over.

The right choice depends on three variables: number of epochs, dataset size relative to available RAM, and whether cold-start latency matters for your workflow.

If you are uncertain, start with OnDemand. It has the lowest downside risk - no large cold start, no RAM commitment - and gives you an accurate baseline. Switch to Pipelined once you confirm you are training for more than a handful of epochs.