We put a vector layer over 5 popular robotics datasets. Here's what that unlocks.

A month ago we wrote about broken normalization stats in public LeRobot datasets. That was rule-based QA — find the deterministic bugs, ship the fixes. It got picked up because it was the right kind of work, but it's not the actual long arc.

The actual long arc is that robotics doesn't have a data platform. Models do. Text does. Code does. Video, audio, sensor data — every other foundation-model modality has a mature platform layer that handles storage, retrieval, indexing, governance, training-time access. Robot trajectories don't. There's a Hugging Face for models, a Snowflake for analytics, a Databricks for everything-on-Spark. There is no equivalent for the demonstrations a humanoid policy is going to learn from.

We're building one. This post is about the first piece — a vector layer over every episode — what we put it on, and what becomes possible the moment it exists.

What we did

We took the five most-trained-on public LeRobot datasets:

• lerobot/pusht
• lerobot/aloha_sim_transfer_cube_scripted
• lerobot/xarm_lift_medium
• unitreerobotics/G1_Dex3_GraspSquare_Dataset
• lerobot/libero

And for each one, ran:

# 1. embed every 5th keyframe → Parquet shard next to the data
python -c "from traceplane.frame_embeddings import compute_frame_embeddings; compute_frame_embeddings('./dataset', stride=5)"

# 2. our existing dataset CI picks up that Parquet
traceplane check ./dataset --ml

The first command samples every 5th keyframe from each episode's video, runs it through DINOv2 ViT-B/14, and writes the resulting 768-dim CLS embeddings to a Parquet shard sitting next to the data. The second command — our existing dataset CI — picks up that Parquet and runs two new checks on top of it: an outlier flag (k-NN cosine distance, z-scored across the dataset) and a near-duplicate cluster check.

That's the whole pipeline. There's no separate vector database, no service to deploy, no GPU at query time. The vector layer is just another column alongside the actions and states, queryable with the same DataFusion that runs vec_cosine_sim over them today.

What the vector layer makes visible

There are two distinct findings in this table, and they're both more interesting than "we found bugs."

A diversity test for scripted-policy datasets

Four of the five datasets above are scripted-policy sim datasets with a single fixed camera. At cosine ≥ 0.95, our embedding layer collapses all (or all-but-one) of the episodes in each into a single visually-indistinguishable bucket — 206/206 for pusht, 50/50 for aloha, 800/800 for xarm, 300/301 for the G1 set. That's not a bug in the threshold. It's the headline finding: scripted-sim datasets are not visually diverse. A policy trained only on them has learned to operate on one scene, not many. When it tries to generalize to a real tabletop, you find out — and you find out the expensive way.

lerobot/libero is the contrast that makes the point land. It's a multi-task dataset with varied scenes — kitchens, study spaces, living rooms — and the same embedding layer at the same threshold produces 10 distinct clusters instead of one, ranging from 454 episodes down to 29. That's actual visual structure: the embedding layer is separating libero's different scenes, not collapsing them into a single blob.

This is one of those measurements you can't make with metadata alone. You need an embedding layer to ask the question "how visually diverse is this dataset actually?"

Distribution tails per dataset

The outlier numbers are the second finding. The top z-scores per dataset are clearly separated from the bulk — pusht's most-distinct episode sits at 4.28σ, libero's at 5.16σ, aloha's at 4.13σ. These aren't a long tail of marginal flags; they're a small set of episodes per dataset that visually do not look like the rest.

We pulled frame strips for a few of them by hand. None of them are broken — pusht ep185 is a valid push with an atypical block-and-target start position; libero ep515 is the same kitchen scene as the rest with different object placement. They're not defects. They're tail episodes — the configurations that are rarest in the dataset.

That's a curation signal, not a defect alarm. It tells you which behaviours your policy will have seen one example of versus a hundred, which is exactly the question that determines whether the policy generalizes or memorizes.

What having this layer unlocks (the bigger arc)

The QA story above is one slice of what a per-frame embedding layer makes possible. The others, in roughly the order we're building them:

1. Semantic search across millions of episodes. Find the 10 episodes most like a query — by an image, by a text prompt (next milestone, SigLIP-based), or by another episode. Robotics teams scroll through hundreds of demos with their eyes today.
2. Retrieval-augmented training. Given a target trajectory, pull its k-NN neighbours from a much larger corpus and condition the policy on them. This is the closest existing analogue to in-context learning in language models.
3. Failure-mode clustering. Take all of a policy's failed rollouts, embed them, cluster. You go from "27% failure rate" to "27% failure rate, broken down into 5 named failure modes."
4. Cross-dataset retrieval. "Show me every grasp of a cylindrical object across every dataset I have access to." Today the answer is "open them one at a time."
5. Foundation-model conditioning. The same embedding space is the natural input to an action codebook / generative trajectory model. You can't build that without the encoder; the encoder is what we just shipped.

None of those are products today. The vector layer is the floor each of them stands on, and the floor is what's new.

A note on QA

Plenty of teams who'd use the platform care about dataset CI more than anything else. So we wired the same vector layer into our existing traceplane check as an opt-in --ml flag. That's how we produced the numbers in the table above — every dataset above passes rule-based QA, but the --ml flag surfaces the diversity and tail-episode signals that rule-based checks can't. That's a real value-add for the QA wedge.

But QA is the entry point. The reason to ship a vector layer is everything else.

We also broke our own QA once during this exercise

Worth flagging, because it's the kind of bug a QA tool ought to be able to catch on itself.

The first time we ran the embedding pipeline on these datasets, it reported zero findings — zero outliers, zero duplicates, perfectly clean. Looked great. The issue: all five datasets are in LeRobot v3 format, which consolidates many episodes into a single video file with per-episode timestamp ranges in the metadata, and our frame-embedding pipeline only knew the v2 layout (one mp4 per episode). It silently saw zero videos, embedded zero keyframes, and produced a perfectly clean report.

A QA tool that produces "PASS" on a broken input is worse than one that crashes. We fixed it (v3 reader is in the package now), re-ran, and the findings above are from the corrected run. We mention it here because the meta-lesson — that "all clean" can be vacuously all-clean — is the same lesson the original audit blog made about rule-based QA on real datasets. Always check whether the check itself is doing anything.

Reproduce it yourself

Everything in this post is in the open-source traceplane package:

pip install 'traceplane[vision]'
python scripts/fetch-audit-datasets.py     # one-time, ~5 GB
python scripts/run-audit-ml.py             # embed + check --ml on all 5 datasets

Or on your own data:

python -c "from traceplane.frame_embeddings import compute_frame_embeddings; compute_frame_embeddings('./your-dataset', stride=5)"
traceplane check ./your-dataset --ml

The per-dataset reports land in traceplane/backend/datasets/audit-ml-reports/*.json and the embedded vectors land next to each dataset's data files as embeddings/frame_embeddings.parquet.

What we're building next

• Text-query search. SigLIP text encoder bolted onto the same vector layer. "Find every demo where the gripper closed on a soft object" as a plain English query.
• Failure-mode clustering UI. Hosted view of grouped failed rollouts so you can scrub through one cluster at a time.
• A real hosted platform. All of the above as a web app, plus the storage / catalog / training / eval pieces we've been building underneath the CLI.

Citations & acknowledgements

• lerobot/pusht, lerobot/aloha_sim_transfer_cube_scripted, lerobot/xarm_lift_medium, lerobot/libero — the LeRobot team at HuggingFace.
• unitreerobotics/G1_Dex3_GraspSquare_Dataset — Unitree Robotics.
• DINOv2 — Meta AI Research.
• Previous post: We audited the most popular public robotics datasets… (April 23, 2026).