Same model, same GPUs, a third of the time
NVIDIA GR00T N1.7 fine-tuned on BridgeV2 for 3,000 steps on 8× A100. knonik streamed every batch from S3; GR00T's stock loader read from local disk. knonik still finished in a third of the time, at a third of the cost.
- 64-66%
- less time to train 3,000 steps
- 62-66%
- lower cost per run
- ≤ 46 s
- to first batch from S3 (stock: 5-12 min)
- 0.05%
- of GPU time waiting on data (stock: 35-60%)
The GPU is only as fast as the data feeding it
knonik is the data layer for robot learning: it stores robot datasets compactly and streams decoded, training-ready batches straight from cloud storage to the GPUs. NVIDIA GR00T N1.7 is a robot foundation model that turns camera images, the robot's state and a language instruction into actions.
In this experiment we fine-tuned GR00T N1.7 on BridgeV2, a large real-world robot manipulation dataset, with knonik and with GR00T's own dataloader. The model, recipe, GPUs and data were the same; only the dataloader changed.
- Model
- NVIDIA GR00T N1.7: Cosmos-Reason2-2B vision-language backbone (frozen), diffusion transformer action head (trained)
- Dataset
- BridgeV2 (nvidia/BridgeData2_LeRobot_v3), 3 cameras at 480×640, 5 fps
- Inputs and outputs
- 3 images, 7-d robot state and a language instruction in; 8-step action chunks out
- GPUs
- 8× NVIDIA A100 40 GB, one machine (96 vCPU, 1.1 TiB RAM)
- Batch size
- 64 per GPU, 512 global
- Training
- 3,000 steps, NVIDIA's stock fine-tune recipe, peak LR 1e-4 with cosine decay
Robot foundation models learn from multi-camera video. Every step of GR00T N1.7 needs 512 fresh samples of three decoded 480×640 frames each, spread across eight GPUs, and if one GPU's batch is late, all eight sit idle until it arrives. The model itself needs about 320 samples/s to stay busy on an A100 machine.
We fine-tuned GR00T N1.7 with NVIDIA's stock recipe and changed only the dataloader. knonik ran at 88-93% of the model's own ceiling over full 3,000-step runs (up to 95% in shorter runs), so the GPUs, not the data path, set the pace. GR00T's stock loader reached about a third of it.
3,000 steps in 1.44 hours, not 4.3
knonik streams from S3. Both stock runs read from local disk.
The stock loader at its default 4 workers per GPU leaves the GPUs waiting 60% of the time. Raising it to 10 workers cuts the waiting to 35% but pushes the host CPU to 98%, and every model step slows from 2.2 to 3.2 seconds, so most of the gain is lost. knonik keeps the GPUs fed with 39% of the CPU, and each step runs at the model's own speed: 1.67 seconds.
Streaming from S3, still 3× faster than local disk
The usual trade-off is that cloud storage is slow, so you copy the dataset onto the training box first. knonik skips that. In these runs every frame came over the network at training time, at 298-299 samples/s: nearly three times what the stock loader managed from local disk.
Nothing is staged up front. The knonik job downloads only model weights and metadata before it starts training. The stock runs first copy the 55 GiB LeRobot corpus to local disk (about 200 s), and then still spend 35-60% of the run waiting on it.
- knonik
- streaming from S3
- 1.44 h
- for 3,000 steps
- 298-299 samples/s
- $56 per run
- Stock GR00T loader
- local disk, best setting
- 4.27 h
- for 3,000 steps
- 104 samples/s
- $165 per run
First batch from S3 in under 46 seconds
Script start to the first training step, 8 GPUs.
From a cold start, every GPU in the knonik runs had its first batch 37-46 seconds after the script launched, read straight from S3, and training started at 70-76 seconds. The stock loader takes 5 to 6 minutes to produce a first batch at 4 workers, and 11.5 to 12 minutes at 10 workers, from local disk.
The stock loader builds 1,024-sample shards by decoding whole episodes for every camera, and every worker on every GPU does this at once before the first step. More workers mean more parallel decodes fighting over the same 96 cores on the 8× A100 machine, so startup gets slower, not faster. knonik fetches and decodes only the frames the first batches need. By step 300 the knonik runs were at 10 minutes, the stock runs at 33-34 minutes.
No stalls in 3,000 steps, not even across the epoch boundary
- 2.7 s
- total wait on data over 3,000 steps, after the first batchstock: 1.4-2.8 hours
- 0.48 s
- longest single wait in the whole runstock: 234-297 s
- 0
- waits over 1 s, on any of the 8 GPUsstock: 3-64 per GPU
- 1.1 ms
- p99 wait per step (median 0.5 ms)stock p99: 34 ms - 30 s
Cumulative time all GPUs spent waiting on data, by training step (first batch included).
The stock loader's waits come as walls, not a trickle. Each worker serves one 1,024-sample shard and prefetches one more; all workers start together, so they run dry together and rebuild their shards at once. At 4 workers that's a wall every 64 steps, 46 of them, median 214 seconds each. At 10 workers it's every 160 steps, 19 walls of about 276 seconds. That's the staircase.
knonik's line stays flat. The median wait per step is half a millisecond, and no GPU waited more than a second even once. The runs cross into a second epoch at step ~2,780 without a pause, and the longest wait in the whole run was still 0.48 seconds.
Two thirds of the stock run isn't training
Splitting each run's steady-state wall clock at knonik's 1.67 s model step: the stock runs spend only 30-34% of their time on model compute. The rest is the GPUs waiting for data, or every step running slower because data loading saturates the CPU the trainer needs.
Same learning curve, 3× sooner
Training loss against hours since the script started. Dots mark the first time each run reaches 0.25.
Per step, the runs learn alike: every run reaches each loss level at about the same step (0.25 at step 1,650-1,660 from S3 and stock alike), and the run-average loss is 0.293-0.294 in all of them.
What changes is the clock. Streaming from S3, knonik reached loss 0.25 after 0.81 hours, against 2.46 and 2.63 hours for the stock loader. Every loss target arrived 3.0-3.4× sooner with knonik.
Same samples, bit for bit
Speed only counts if the model sees the same data. A parity check over the full corpus compared knonik's output against the stock loader's: decoded frames are bit-exact, state, action and language match exactly, GR00T's processor and collator produce the same tensors, and each epoch covers the same set of samples.
The only difference is order, by design: the stock loader serves shards built from slices of episodes, while knonik shuffles across the whole dataset. Training code, optimiser, augmentation and learning-rate schedule are NVIDIA's, unchanged.
Faster, cheaper, more efficient
| Metric | knonikstreaming from S3 | knoniklocal disk | Stock, 10 workerslocal disk | Stock, 4 workerslocal disk |
|---|---|---|---|---|
| Throughput (samples/s) | 298-299 | 282-292 | 104 | 91 |
| Time to train 3,000 steps | 1.44-1.45 h | 1.47-1.53 h | 4.27 h | 4.73 h |
| Script start → first step | 70-76 s | 65-66 s | 801 s | 372 s |
| First batch on every GPU | 37-46 s | 28-39 s | 690-723 s | 307-330 s |
| Time to reach loss 0.25 | 0.81 h | 0.71-0.74 h | 2.46 h | 2.63 h |
| Total wait on data (after first batch) | 2.7-3.1 s | 2.4-2.7 s | 5,129 s | 9,922 s |
| Longest single wait | 0.48-0.49 s | 0.19-0.64 s | 297 s | 234 s |
| Step time, p99 | 2.07-2.10 s | 2.31-2.43 s | 33.7 s | 36.6 s |
| Model step, median | 1.67 s | 1.68-1.74 s | 3.16 s | 2.23 s |
| Host CPU used (of 96 vCPU) | 39% | 37-39% | 98% | 88% |
| Cost of the run (on-demand) | $56-57 | $61-63 | $165 | $182 |
Cost is billed instance time at the AWS on-demand list price of $37.688/h.