Knonik Processing

This guide explains how to use the Knonik processing agent from the wheel-based Knonik package. Unlike ingest and the dataloader (CLI / Python APIs), processing is a local web dashboard: you launch it with knonik viz and work in your browser. It lets you visualize, quality check, auto-annotate, manually annotate, and compare the episode datasets produced by knonik ingest.

1. What The Processing Agent Does

The processing agent operates on ingested episode datasets — the per-episode directories knonik ingest writes (each with metadata.json, RGB/depth/numeric streams, and an annotations/ folder) and not the final packed sharded dataset. You point the dashboard at such a directory and then:

• ▸Visualize — play episode video, scrub a timeline, and plot numeric streams (state / action / qpos / …) and trajectories.
• ▸Quality check — score episodes against quality heuristics.
• ▸Manual/Auto-annotate — manually/automatically segment episodes into sub-steps and generate language labels and object masks.
• ▸Compare — overlay trajectories across episodes, and open several datasets side by side.

Some capabilities are gated by your account (see section 12): manual annotation and visualization are always available; quality check and auto-annotation are enabled per account.

The annotations and quality reports you produce live next to the episodes, so a later knonik pack picks them up for training.

2. Install

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install /path/to/knonik-0.1.0-cp311-cp311-manylinux*.whl   # match your Python

Verify:

knonik --version

Auto-annotation extras. Segment language labels use a Vision-Language Model. You can point it at a hosted provider (OpenAI / Anthropic) using your own API key — no extra install. Object masking runs locally.

3. Log In

Log in once with an account entitled for processing:

knonik login --product processing

On headless Linux machines without an OS keyring:

knonik login --product processing --allow-file-key-store

Check status / log out:

knonik status --product processing
knonik logout --product processing

knonik status --product processing also shows which features your account has (manual annotation, auto-annotation, quality gate, session limit).

4. Launch The Dashboard

knonik viz                      # serves on http://127.0.0.1:3000 and opens a browser
knonik viz --port 8080          # choose a different port
knonik viz --no-browser         # print the URL instead of opening a browser

Working over SSH (remote box)

The dashboard is loopback-only, so forward the port from your laptop:

ssh -L 3000:127.0.0.1:3000 user@remote-box
# on the remote box:
knonik viz --no-browser
# then open the printed http://127.0.0.1:3000/#knonik-bootstrap=... URL in your local browser

5. Open A Dataset (Create A Session)

In the dashboard you create a session by pointing it at a dataset root — the directory that directly contains the episode_000000/, episode_000001/, … folders from knonik ingest

knonik_thread_velcro/            ← point the session here (dataset root)
├── config.json                  ← dataset config, written by ingest (see §6)
├── episode_000000/
│   ├── metadata.json
│   ├── rgb_*.webm
│   ├── <numeric_stream>/        (zarr arrays)
│   └── annotations/             (segments / language / objects written here)
├── episode_000001/
└── …

Notes:

• ▸You can open several sessions at once (one per dataset) to compare them; the session list shows all open datasets. Your account's session limit (max_sessions) caps how many can be open simultaneously.
• ▸Closing a session just detaches the dashboard; it never deletes your data.
• ▸For Quality check, the dataset root must contain a config.json — see section 6 for how to write it.

6. Dataset Config

This is not the ingest config. The ingest config tells knonik ingest how to read your raw data. This dataset config describes the already-ingested episodes — which streams exist and how to interpret the numeric channels (arms, degrees of freedom, action semantics). It lives at the dataset root as config.json.

It is used by:

• ▸Quality check (section 8) — reads the stream layout, the arm slicing, and the per-channel semantics from it; the run is rejected without it, and an underspecified config makes quality skip checks rather than fail (see warnings).
• ▸Visualization — for the plot frame rate, channel labels, per-arm grouping, and trajectories.
• ▸Auto-segmentation — reads numeric_correspondance.qpos to load the state signal it segments on.

6.1 Ingest Writes It For You

You do not write this file from scratch. Every knonik ingest run ends by writing config.json into the output directory, filled in as far as the data allows and marked up where it cannot go further:

Wrote dataset config /data/my_dataset/config.json
  8 value(s) need you: see "_needs_input" in that file, or run
  'knonik config check /data/my_dataset'. Quality check will refuse to run
  until they are filled.

An existing config.json is never overwritten — a config you tuned by hand outlives the run that produced the data. Pass --force-config (or force_config=True from the Python API) when you do want it regenerated:

knonik ingest --config ingest.json --input /raw --output /data/my_dataset --force-config

For datasets you ingested before this existed, generate one in place. It reads only what is already on disk, so nothing is re-ingested:

knonik config init /data/my_dataset

What it fills in, and what it refuses to

The generator writes a value only when it is declared — by the source format, a URDF, a media container header, or the episode metadata — or measured from the data with an exact test. It never infers meaning from a name, a channel position, or a range of values. That line is deliberate: a config that is 80% written and honest about the rest is worth more than one that is 100% written and wrong in the two fields that decide your scores.

Per-channel names come from ROS JointState.name and JointTrajectory.joint_names, and from LeRobot's features[...].names. RLDS and TFRecord sources declare no channel names, so those datasets have more to fill in.

fps gets special care, because two sources can disagree: the timestamps say what the data actually does, the container header says what the encoder was configured with. When timestamps are real seconds they win; when they are frame indices (DROID ingests this way) they carry no rate at all and the container header is used; when both exist and differ by more than 2%, the measured value is written and the disagreement is recorded in _needs_input for you to settle.

Filling in the blanks

Unfilled values are <FILL:...> tokens sitting at the key they belong to, so you edit in place:

"numeric_correspondance": {
  "action": "<FILL:stream_name>",
  "qpos": "<FILL:stream_name|omit>"
},
"action_values": { "type": "<FILL:absolute|delta>" },
"arms": {
  "<FILL:arm_name>": {
    "indices": "<FILL:[start,end]>",
    "dof_names": "<FILL:[names,...]|omit>",
    "gripper_dim": "<FILL:index_within_slice|omit>"
  }
},
"arm_names": ["<FILL:arm_name>"]

Each one also has an entry in the _needs_input block at the bottom of the file, carrying the question, why the generator would not answer it, and the evidence it collected — the index-annotated channel list, the candidate streams and their widths, the ranges over which two streams are known to describe the same channels. _needs_input and _generated are ignored by every tool that reads the config; delete them once you are done, or leave them.

To see what is outstanding without opening the file:

knonik config check /data/my_dataset

/data/my_dataset/config.json

8 value(s) still need you:

  numeric_correspondance.action
      Which numeric stream holds the commanded / target signal?
      not filled in: The source format did not declare an action stream. Topic and
      array names are not evidence — a stream called *_command is not necessarily
      the one you train on.
      evidence: { "candidates": [ … stream names, widths, channel names … ] }
  …

Worth reviewing, but the config is usable without them:

  streams.rgb.fps
      Confirm the capture rate: the two sources disagree.
  numeric_dimensions.*.bounds
      Optional: give each non-gripper channel [min, max] from your robot's URDF.

Quality check will refuse to run until the blocking values are filled.

Two classes of entry: the ones that block (a placeholder is sitting in the file waiting for you) and the ones that are advisory (nothing was left blank — these are review notes, like an fps disagreement or a stream that only some episodes have). Exit status is non-zero while anything blocks, which makes check usable as a pipeline gate. Once nothing blocks, it runs the same validation quality itself applies, so a config that passes check really loads.

Some values only become derivable after you answer the semantic ones — answering one question unlocks the next:

knonik config resolve /data/my_dataset                     # e.g. dof_names, once indices are set
knonik config resolve /data/my_dataset --with-dimensions   # also scaffold numeric_dimensions

resolve never decides anything semantic; it only propagates your own answers, and it prunes the questions you have already answered so the file keeps reflecting what is actually left. --with-dimensions is opt-in because the scaffold it writes introduces new values you must fill.

Quality refuses to run while any <FILL:...> token remains, and names the outstanding keys with their questions — from the CLI and from the dashboard alike. A half-finished config never silently scores your data.

That is the whole workflow. The rest of this section documents what each field means, for when you need to write or correct one by hand.

6.2 Required Blocks

A quality run fails outright if any of these five are missing. Everything else is optional.

streams

Names must match the episode directory names exactly — copy them from metadata.json, do not retype them:

numeric_correspondance

Maps the two logical roles quality understands onto your stream names:

"numeric_correspondance": { "action": "actions", "qpos": "qpos" }

• ▸action is required — it is the stream every command-side check runs on.
• ▸qpos (the measured state) is optional. Omit it and the motion-side checks (jerky_motion, abrupt_motion, mostly_static_motion, invalid_motion) are simply not run; the command-side ones still are.
• ▸Each stream also needs its timestamp array. Quality looks for <stream>_time next to it — exactly what ingest writes — unless you override it under numeric_timestamps.

arms and arm_names

An arm is a contiguous slice of the numeric vector. This is where most configs go wrong, so read the rules carefully:

arm_names fixes the order and is what quality iterates over — a report is produced per arm. Channels not covered by any arm still plot (grouped as "other") but are not quality-checked.

A single-arm robot is just one entry:

"arms": {
  "right": {"dof": 6, "dof_names": ["j1","j2","j3","j4","j5","j6","gripper"],
            "indices": [0, 6], "gripper_dim": 6}
},
"arm_names": ["right"]

A 14-channel bimanual vector splits into two inclusive halves — [0, 6] and [7, 13].

6.3 Optional Blocks

numeric_dimensions

Per-channel semantics. Omitting it is legal but costs you checks: quality raises warning 700 and skips saturation and correct angular handling.

"numeric_dimensions": {
  "action": [
    {"name": "x", "kind": "linear", "bounds": [-1.0, 1.0]},
    {"name": "yaw", "kind": "angular", "period": 6.283185307179586},
    {"name": "gripper", "kind": "gripper"}
  ]
}

Rules that are easy to get wrong:

• ▸The list covers the entire numeric vector, not one arm. A 14-channel bimanual action stream needs 14 entries; quality slices out each arm's part using that arm's indices.
• ▸Key it by either the logical role ("action", "state") or your actual stream name ("actions", "qpos") — both resolve.
• ▸kind must be one of linear, angular, gripper, other. Anything else is a hard error.
• ▸angular accepts period (default 2π) so wrap-around is not read as a huge jump.
• ▸bounds is [min, max] with max > min. Saturation is only checked when every non-gripper channel of that arm has bounds (or the arm sets action_bounds); otherwise you get warning 702.

action_values

"action_values": { "type": "absolute" }

absolute (default) means an action is a target pose/position; delta means it is an increment on the current state. Quality interprets motion differently for each, so a delta-action dataset labelled absolute will misreport. Only those two values are accepted.

numeric_timestamps

Only needed when your timestamp arrays are not the <stream>_time that ingest writes, or when the values are frame indices rather than seconds:

"numeric_timestamps": {
  "action": {"stream": "action_ts", "semantics": "seconds"},
  "state":  {"semantics": "frame_index"}
}

Keys must be action and/or state. semantics is auto (default), seconds, or frame_index.

server and agent_meta

Free-form dicts passed through to the quality report — use agent_meta to stamp a run with your own labels (operator, capture site, robot serial). Neither affects scoring.

6.4 Examples

Every config below is a real one, written against a real ingested dataset. Each example names the source format it was ingested from, because that is what decides the stream names you have to work with.

A. LeRobot v3 → bimanual ALOHA (14 channels, 2 arms)

A LeRobot v3 dataset ingested with the default layout. LeRobot names its numeric features state and action, and its cameras keep their LeRobot keys (cam_high, cam_low, cam_left_wrist, cam_right_wrist) — so the stream names come out clean and the config is mostly about the arm split.

The action/state vector is 14 wide: seven channels per arm, gripper last. That splits into two inclusive halves, [0, 6] and [7, 13], and in both cases the gripper is the 7th channel of the slice — gripper_dim: 6, relative to the slice, not channel 6 and 13 of the full vector.

{
  "dataset_root": "/data/output_v3",
  "streams": {
    "numeric": {
      "names": ["state", "action"]
    },
    "rgb": {
      "names": ["cam_high", "cam_low", "cam_left_wrist", "cam_right_wrist"],
      "fps": 50
    },
    "depth": {
      "names": [],
      "fps": 50
    }
  },
  "numeric_correspondance": {
    "action": "action",
    "qpos": "state"
  },
  "arms": {
    "left": {
      "dof": 6,
      "dof_names": ["left_waist", "left_shoulder", "left_elbow", "left_forearm_roll",
                    "left_wrist_angle", "left_wrist_rotate", "left_gripper"],
      "indices": [0, 6],
      "gripper_dim": 6
    },
    "right": {
      "dof": 6,
      "dof_names": ["right_waist", "right_shoulder", "right_elbow", "right_forearm_roll",
                    "right_wrist_angle", "right_wrist_rotate", "right_gripper"],
      "indices": [7, 13],
      "gripper_dim": 6
    }
  },
  "arm_names": ["left", "right"]
}

This config runs, but it declares no numeric_dimensions, so quality reports warning 700 and skips saturation and angular-wrap handling. Adding the block turns those checks back on — one entry per channel of the whole 14-wide vector, in channel order:

"action_values": { "type": "absolute" },
"numeric_dimensions": {
  "action": [
    {"name": "left_waist", "kind": "angular", "bounds": [-3.14, 3.14]},
    {"name": "left_shoulder", "kind": "angular", "bounds": [-1.85, 1.26]},
    {"name": "left_elbow", "kind": "angular", "bounds": [-1.76, 1.6]},
    {"name": "left_forearm_roll", "kind": "angular", "bounds": [-3.14, 3.14]},
    {"name": "left_wrist_angle", "kind": "angular", "bounds": [-1.8, 2.23]},
    {"name": "left_wrist_rotate", "kind": "angular", "bounds": [-3.14, 3.14]},
    {"name": "left_gripper", "kind": "gripper"},
    {"name": "right_waist", "kind": "angular", "bounds": [-3.14, 3.14]},
    {"name": "right_shoulder", "kind": "angular", "bounds": [-1.85, 1.26]},
    {"name": "right_elbow", "kind": "angular", "bounds": [-1.76, 1.6]},
    {"name": "right_forearm_roll", "kind": "angular", "bounds": [-3.14, 3.14]},
    {"name": "right_wrist_angle", "kind": "angular", "bounds": [-1.8, 2.23]},
    {"name": "right_wrist_rotate", "kind": "angular", "bounds": [-3.14, 3.14]},
    {"name": "right_gripper", "kind": "gripper"}
  ]
}

Take the bounds from your robot's URDF joint limits, not from the observed data — bounds fitted to the data can never report saturation. Repeat the same list under "state" if the state vector has the same layout (for LeRobot ALOHA it does).

B. TFRecord / RLDS → DROID (7-DoF end-effector, 1 arm)

A DROID shard ingested from TFRecord. RLDS-style sources give you state and action as flat vectors, and the cameras keep their RLDS observation keys — hence names like exterior_image_1_left.

The interesting part is that this action space is Cartesian, not joint: xyz plus roll/pitch/yaw plus gripper. That is exactly what numeric_dimensions is for — the three rotational channels are declared angular so a wrap from +π to −π is not scored as a huge jump, and the gripper is excluded from the smoothness and saturation maths:

{
  "dataset_root": "/data/droid_ingest_lng",
  "streams": {
    "numeric": {
      "names": ["state", "action"],
      "channels": 2,
      "compressor": "zstd",
      "level": 3
    },
    "rgb": {
      "names": ["exterior_image_1_left", "exterior_image_2_left", "wrist_image_left"],
      "codec": "vp9",
      "fps": 15
    },
    "depth": {
      "names": [],
      "codec": "ffv1",
      "fps": 15,
      "quant": 1
    }
  },
  "numeric_correspondance": {
    "action": "action",
    "qpos": "state"
  },
  "numeric_dimensions": {
    "action": [
      {"name": "x", "kind": "linear"},
      {"name": "y", "kind": "linear"},
      {"name": "z", "kind": "linear"},
      {"name": "roll", "kind": "angular"},
      {"name": "pitch", "kind": "angular"},
      {"name": "yaw", "kind": "angular"},
      {"name": "gripper", "kind": "gripper"}
    ],
    "state": [
      {"name": "x", "kind": "linear"},
      {"name": "y", "kind": "linear"},
      {"name": "z", "kind": "linear"},
      {"name": "roll", "kind": "angular"},
      {"name": "pitch", "kind": "angular"},
      {"name": "yaw", "kind": "angular"},
      {"name": "gripper", "kind": "gripper"}
    ]
  },
  "action_values": {
    "type": "absolute"
  },
  "arms": {
    "right": {
      "dof": 6,
      "dof_names": ["j1", "j2", "j3", "j4", "j5", "j6", "gripper"],
      "indices": [0, 6],
      "gripper_dim": 6
    }
  },
  "arm_names": ["right"]
}

C. HDF5 / Zarr episodic → bimanual, minimal

The smallest config that still checks two arms properly. Streams, role mapping, arms — codecs and levels left at ingest defaults. Note qvel is ingested and plots in the dashboard, but is not one of the two roles quality scores, so it is simply not mentioned in numeric_correspondance:

{
  "dataset_root": "/data/knonik_bimanual_transfer",
  "streams": {
    "numeric": {
      "names": ["actions", "qpos", "qvel"]
    },
    "rgb": {
      "names": ["top"],
      "fps": 50
    },
    "depth": {
      "names": [],
      "fps": 50
    }
  },
  "numeric_correspondance": {
    "action": "actions",
    "qpos": "qpos"
  },
  "arms": {
    "left": {
      "dof": 6,
      "dof_names": ["j1", "j2", "j3", "j4", "j5", "j6", "gripper"],
      "indices": [0, 6],
      "gripper_dim": 6
    },
    "right": {
      "dof": 6,
      "dof_names": ["j1", "j2", "j3", "j4", "j5", "j6", "gripper"],
      "indices": [7, 13],
      "gripper_dim": 6
    }
  },
  "arm_names": ["left", "right"]
}

D. ROS bag / MCAP

ROS-ingested datasets are different enough to need their own section — see 6.5.

E. Smallest config that runs

Every field here is required; nothing is decorative. Expect warning 700 (no numeric_dimensions) and 702 (no bounds, saturation skipped):

{
  "dataset_root": "/data/my_dataset",
  "streams": {
    "numeric": {"names": ["action", "qpos"]},
    "rgb": {"names": ["cam_top"], "fps": 30},
    "depth": {"names": []}
  },
  "numeric_correspondance": {"action": "action", "qpos": "qpos"},
  "arms": {"arm": {"dof": 6, "dof_names": ["j1","j2","j3","j4","j5","j6","gripper"],
                   "indices": [0, 6], "gripper_dim": 6}},
  "arm_names": ["arm"]
}

6.5 ROS Bags and MCAP

ROS-ingested datasets are the ones people get wrong, because ROS has no state and no action vector. ROS ingest keeps one numeric stream per topic, named after the sanitized topic path, and never concatenates them. So the config has to do the job the recording never did: say which topic is the command, which is the measurement, and which channels of each are the arm. dir_type: "mcap" is the same reader — everything here applies unchanged to .mcap and .db3 bags.

Step 1 — read the widths off the episode

metadata.json records every stream and its channel width under ros_numeric_streams, which is what you need to write indices:

python -c "import json;m=json.load(open('/data/rosbag/episodes/episode_000000/metadata.json'));print(json.dumps({k:m[k] for k in ('rgb_streams','depth_streams','ros_numeric_streams','episode_length')},indent=2))"

For a Toyota HSR bag that prints:

{
  "rgb_streams": ["hsrb_hand_camera", "hsrb_head_rgbd_sensor"],
  "depth_streams": ["hsrb_head_rgbd_sensor_image_rect_raw_compressedDepth"],
  "ros_numeric_streams": {
    "hsrb_arm_trajectory_controller_command": 5,
    "hsrb_base_scan": 963,
    "hsrb_command_velocity": 6,
    "hsrb_gripper_controller_command": 1,
    "hsrb_head_trajectory_controller_command": 2,
    "hsrb_joint_states": 13,
    "hsrb_odom": 13,
    "hsrb_servo_states": 33,
    "hsrb_wrist_wrench_raw": 6
  },
  "episode_length": 4221
}

Depth stream names keep the full topic path including the encoding suffix (…_image_rect_raw_compressedDepth). Copy them verbatim; do not tidy them up.

Step 2 — get fps from the timestamps, not from metadata

start_time / end_time in the metadata cover the whole bag, including any lead-in before the first camera frame, so dividing by them under-reports the rate. Use the timestamp array of the stream you care about:

python -c "import zarr,numpy as np;t=np.asarray(zarr.open('/data/rosbag/episodes/episode_000000/hsrb_joint_states_time','r')[:]);print((len(t)-1)/(t[-1]-t[0]))"
# 29.84  ->  "fps": 30

Step 3 — pick the two roles

Leave everything else out of numeric_correspondance. odom, base_scan, servo_states, wrist_wrench_raw still plot in the dashboard — they are just not what quality scores.

Step 4 — the trap: one indices slices both streams

This is where ROS configs break. arms.<name>.indices is applied to the action stream and to the state stream, with the same numbers. On a LeRobot or RLDS dataset that is harmless because both vectors share a layout. On a ROS bag they almost never do — the command topic carries only the controller's joints, while joint_states carries every joint on the robot, alphabetically:

indices: [0, 4] would silently compare the commanded wrist against the drive wheels. Nothing errors; the numbers just become meaningless.

You usually do not have to work this out by hand. ROS ingest records JointState.name and JointTrajectory.joint_names into each episode's metadata.json under channel_names. Name the two streams in the generated config and then ask for the consequences:

# after filling in numeric_correspondance.action and .qpos
knonik config resolve /data/rosbag/episodes

The arms entry in _needs_input then carries an index-annotated channel list for both streams, plus aligned_ranges — the inclusive ranges over which they name the same channel:

"aligned_ranges": [[0, 2]],
"action_channels": {"width": 5, "channels": [
  {"index": 0, "name": "arm_flex_joint"},
  {"index": 1, "name": "arm_lift_joint"},
  {"index": 2, "name": "arm_roll_joint"},
  {"index": 3, "name": "wrist_flex_joint"},
  {"index": 4, "name": "wrist_roll_joint"}
]}

[[0, 2]] is computed from the joint names alone — no correlation of values, no assumption about what an arm is. It tells you where a slice may go; which of those channels you call an arm is still yours to say.

For bags recorded without joint names, verify the alignment from the data before you trust a slice:

python - <<'PY'
import zarr, numpy as np
E = "/data/rosbag/episodes/episode_000000"
a = np.asarray(zarr.open(f"{E}/hsrb_arm_trajectory_controller_command", "r")[:])
s = np.asarray(zarr.open(f"{E}/hsrb_joint_states", "r")[:])
for i in range(a.shape[1]):
    err = [np.abs(s[:, j] - a[:, i]).mean() for j in range(s.shape[1])]
    print(f"action ch {i} tracks state ch {int(np.argmin(err))}  (mae {min(err):.3f})")
PY
# action ch 0 tracks state ch 0   -> aligned
# action ch 1 tracks state ch 1   -> aligned
# action ch 2 tracks state ch 2   -> aligned
# action ch 3 tracks state ch 11  -> NOT aligned
# action ch 4 tracks state ch 12  -> NOT aligned

You have two honest ways out, and the choice is a real trade-off:

Do not "fix" the mismatch by re-slicing one stream on disk — quality reads the Zarr arrays as ingest wrote them.

Step 5 — grippers on their own topic

ROS usually publishes the gripper as a separate topic (hsrb_gripper_controller_command, 1 channel). It is not inside the arm slice, so omit gripper_dim entirely and give no channel "kind": "gripper" in the arm's range. Quality then runs the arm checks with no gripper channel to exclude, which is correct. Setting gripper_dim to a channel that is really a joint makes quality drop a real joint from the smoothness maths.

Once indices is set, the labels follow from the same names — knonik config resolve fills dof_names for you, and knonik config resolve --with-dimensions scaffolds a full-width numeric_dimensions block with every channel named and every kind left for you to set.

The full config

This is the validated config for the HSR bag above — it runs clean, with zero config warnings:

{
  "dataset_root": "/data/rosbag/episodes",
  "streams": {
    "numeric": {
      "names": [
        "hsrb_joint_states",
        "hsrb_arm_trajectory_controller_command",
        "hsrb_gripper_controller_command",
        "hsrb_head_trajectory_controller_command",
        "hsrb_command_velocity",
        "hsrb_odom",
        "hsrb_servo_states",
        "hsrb_wrist_wrench_raw",
        "hsrb_base_scan"
      ],
      "compressor": "zstd",
      "level": 3,
      "fps": 30
    },
    "rgb": {
      "names": ["hsrb_hand_camera", "hsrb_head_rgbd_sensor"],
      "codec": "vp9",
      "fps": 30
    },
    "depth": {
      "names": ["hsrb_head_rgbd_sensor_image_rect_raw_compressedDepth"],
      "codec": "ffv1",
      "fps": 30
    }
  },
  "numeric_correspondance": {
    "action": "hsrb_arm_trajectory_controller_command",
    "qpos": "hsrb_joint_states"
  },
  "numeric_timestamps": {
    "action": {"stream": "hsrb_arm_trajectory_controller_command_time", "semantics": "seconds"},
    "state": {"stream": "hsrb_joint_states_time", "semantics": "seconds"}
  },
  "action_values": { "type": "absolute" },
  "numeric_dimensions": {
    "action": [
      {"name": "arm_flex_joint", "kind": "angular", "bounds": [-2.62, 0.17]},
      {"name": "arm_lift_joint", "kind": "linear", "bounds": [-0.05, 0.69]},
      {"name": "arm_roll_joint", "kind": "angular", "bounds": [-2.09, 3.84]},
      {"name": "wrist_flex_joint", "kind": "angular", "bounds": [-1.92, 1.22]},
      {"name": "wrist_roll_joint", "kind": "angular", "bounds": [-1.92, 3.67]}
    ],
    "state": [
      {"name": "arm_flex_joint", "kind": "angular", "bounds": [-2.62, 0.17]},
      {"name": "arm_lift_joint", "kind": "linear", "bounds": [-0.05, 0.69]},
      {"name": "arm_roll_joint", "kind": "angular", "bounds": [-2.09, 3.84]},
      {"name": "base_l_drive_wheel_joint", "kind": "angular"},
      {"name": "base_r_drive_wheel_joint", "kind": "angular"},
      {"name": "base_roll_joint", "kind": "angular"},
      {"name": "hand_l_spring_proximal_joint", "kind": "angular"},
      {"name": "hand_motor_joint", "kind": "gripper"},
      {"name": "hand_r_spring_proximal_joint", "kind": "angular"},
      {"name": "head_pan_joint", "kind": "angular"},
      {"name": "head_tilt_joint", "kind": "angular"},
      {"name": "wrist_flex_joint", "kind": "angular", "bounds": [-1.92, 1.22]},
      {"name": "wrist_roll_joint", "kind": "angular", "bounds": [-1.92, 3.67]}
    ]
  },
  "arms": {
    "arm": {
      "dof": 3,
      "dof_names": ["arm_flex_joint", "arm_lift_joint", "arm_roll_joint"],
      "indices": [0, 2]
    }
  },
  "arm_names": ["arm"],
  "agent_meta": {
    "robot": "Toyota HSR",
    "source_format": "rosbag"
  }
}

Points worth copying:

• ▸numeric_dimensions lists every channel of each stream — 5 for the command, 13 for joint_states — even though the arm only uses 0–2. Quality slices the list with the same indices, so the list must be full length.
• ▸bounds come from the HSR URDF joint limits, padded slightly for encoder noise so normal motion is not reported as saturation. arm_lift_joint is prismatic → "kind": "linear"; every other arm joint is revolute → "kind": "angular".
• ▸hand_motor_joint is tagged gripper in the state list for the dashboard's benefit, but it sits at channel 7 — outside the [0, 2] arm slice — so it plays no part in scoring.
• ▸numeric_timestamps is spelled out. ROS ingest writes <stream>_time next to each stream, so it would be found anyway; being explicit documents that these are seconds from episode start, not frame indices.

The command-side-only variant is the same file with qpos and the state timestamp entry removed, and:

"arms": {
  "arm": {
    "dof": 5,
    "dof_names": ["arm_flex_joint", "arm_lift_joint", "arm_roll_joint",
                  "wrist_flex_joint", "wrist_roll_joint"],
    "indices": [0, 4]
  }
},
"arm_names": ["arm"]

ROS-specific mistakes

6.6 Config Warnings

A quality run reports config-level warnings alongside the per-episode flags. They are not failures — they tell you which checks were skipped because the config did not say enough:

6.7 Common Mistakes

7. Visualize Episodes

Once a session is open, pick an episode to:

• ▸Play the RGB video and scrub a frame-accurate timeline.
• ▸Plot numeric streams — state, action, qpos/qvel, etc. — over time.
• ▸View trajectories for the episode.
• ▸See segments (sub-step boundaries) overlaid on the timeline once the episode has been segmented or annotated.

Visualization is always available (no entitlement required) and reads straight from the episode directory — nothing is modified.

8. Quality Check

Requires the quality entitlement on your account.

Quality scans each episode for common data problems and produces a report you can sort/filter by, so you can drop or fix bad episodes before training.

How to run it:

1. ▸Make sure the dataset root contains the dataset config.json (quality reads the stream layout from it; without it the run is rejected). See section 6.
2. ▸In the session, start a Quality run and choose a profile:
   • ▸conservative (default) — flags only clear problems.
   • ▸standard — balanced.
   • ▸strict — flags more aggressively.
3. ▸The agent writes quality_metrics.json (per-episode scores, per arm) and quality_flagged.json (just the flagged episodes) into the dataset root, and shows the flags in the UI.

Typical flags include: dead_actions, idle_action, jerky_commands / jerky_motion, abrupt_commands / abrupt_motion, gripper_chatter, saturated, mostly_static_commands / mostly_static_motion, low_entropy / high_entropy, invalid_commands / invalid_motion, insufficient_frames, and irregular_timestamps / invalid_timestamps.

The report also carries the config-level warnings from section 6.6 — check those first if a flag you expected never appears.

Re-running quality overwrites both files. The report stays in the dataset root, so it is available the next time you open the session and to any downstream tooling.

9. Auto-Annotate

Not available yet — activating shortly. Auto-annotation is documented here so you know what's coming, but it is not yet serving production runs. Everything else in this guide, including the quality gate, works today. Pro subscribers get auto-annotation the moment it switches on, at no extra cost.

Requires the auto-annotation entitlement on your account.

Auto-annotation turns raw episodes into labeled, segmented data with minimal manual work:

• ▸Temporal segmentation — split an episode into sub-steps. Segment a single episode, run segment-all as a batch job over the whole dataset (with a job you can poll), or refine an existing segmentation.
• ▸Language labels (VLM) — generate a natural-language description per segment using a Vision-Language Model. You select the provider and supply your own API key in the dashboard; the agent sends the relevant frames/prompt to that provider and stores the returned text as the segment's description. Knonik also supports local VLM for annotation, make your choice in the auto-annotate dropbox.
• ▸Object masks (SAM) — detect and mask objects in a frame, including from a text prompt, and keep the accepted masks as object annotations. SAM runs locally.

Auto-annotation is a starting point: every result lands as an editable draft you review in the manual step below.

VLM data flow. Object masking (SAM) and segmentation run locally. Language labeling sends the frames and prompt for the chosen segments to the VLM provider you configure (e.g. OpenAI / Anthropic) under your own API key. If that is not acceptable for your data, skip VLM labeling and write descriptions manually (section 10).

To run with different model providers, install corresponding package, for example to use openai api key

pip install openai

10. Manually Annotate

Always available (the manual annotation feature is on by default).

The human-in-the-loop editor lets you produce or correct annotations by hand:

• ▸Edit segment boundaries — adjust where each sub-step starts and ends on the timeline, add or remove segments.
• ▸Write / edit language descriptions for episodes and segments.
• ▸Select and accept objects — draw or text-prompt object masks and accept the ones you want.
• ▸Accept — accepted edits are committed to the episode's annotations/ folder; drafts you don't accept are not.

This is the same editor you use to review auto-annotation output: auto-generate first, then fix up and accept.

11. Compare Episodes & Datasets

• ▸Within a dataset — overlay trajectories across episodes to spot outliers or inconsistent demonstrations, and step through episodes in the same view.
• ▸Across datasets — open one session per dataset root and switch between (or view side by side) to compare two captures, two robots, or pre/post-processing versions of the same data.

Comparison is read-only visualization; it never changes your episodes.

12. Feature Entitlements

What you can do is controlled by your account's features (shown by knonik status --product processing):

If a panel is greyed out or an action returns feature_not_enabled, your account lacks that entitlement — ask your admin to enable it.

13. Where Results Are Saved

• ▸Annotations (segments, language, objects you accept) are written into each episode's annotations/ directory, alongside the streams.
• ▸Quality report is written as quality_metrics.json (with quality_flagged.json listing just the flagged episodes) in the dataset root.

Because these live with the episodes, a later knonik pack includes them, so the labels and segment information flow through to the KHLP dataset your training job loads.