Physical AI

Quick facts

Topic

Physical AI

Audience

Procurement leads, ML ops, robotics engineers

Deliverable

Buyer-facing reference + procurement guidance

What physical AI needs from training data

Physical AI models learn from multi-modal temporal sequences: state-action-observation tuples recorded at task-relevant frequencies. Google's RT-1 trained on 130,000 demonstrations across 700 tasks, each episode pairing 3 Hz RGB with 7-DOF end-effector actions^[1]. DROID scales to 76,000 trajectories from 564 scenes and 84 tasks, the diversity a generalist policy needs^[2].

Volume is the easy part. What separates trainable data from expensive noise is calibration, timing, and provenance you can verify (thresholds below). Open X-Embodiment aggregated 60 datasets and over 1 million trajectories, then spent most of its effort unifying coordinate frames and action spaces after collection^[3]. Check those prerequisites before a training run, not after.

On Truelabel's physical AI marketplace, buyers post a spec and matched suppliers return sample packets with QA evidence before any commitment to scale. Deliveries are rights-cleared, carrying contributor consent artifacts and per-trajectory provenance, in RLDS, LeRobot, MCAP, or a custom schema.

Vision-language-action models and the embodiment gap

RT-2 and OpenVLA treat robot actions as discrete tokens in an autoregressive sequence, which lets a vision-language backbone trained on the web transfer to control^[4]. RT-2 reached 62% success on unseen tasks on top of PaLI-X's 55-billion-parameter backbone^[4]. OpenVLA reproduces the recipe with discretized action tokens, not a diffusion head^[5].

The catch is the embodiment gap. A policy trained on a Franka Panda with a parallel-jaw gripper will not drop onto a UR5e with a suction cup without retraining or adapter layers. Open X-Embodiment's RT-X models attacked this by training across 22 robot morphologies at once, learning a shared representation^[3], yet policies transferred across embodiments still lose success relative to a single-robot specialist unless retrained or adapted. That gap, not model size, is usually what stops a bought dataset from working on your arm.

Hugging Face's LeRobot gives you embodiment-agnostic loaders and policy code, which cuts the plumbing for multi-robot training. When you screen a candidate dataset, count embodiments: three or more platforms with overlapping tasks pretrains a more robust VLA than any single-platform set.

World foundation models: from video prediction to physics

World foundation models learn forward dynamics, predicting future states from current observations and actions, which enables model-based planning, synthetic data generation, and counterfactual rollouts. NVIDIA Cosmos trains video diffusion transformers on 20 million hours of driving footage to generate physically plausible multi-camera sequences for AV validation^[6]. GR00T N1 pretrains its manipulation world model on a heterogeneous mixture of real-robot trajectories, human video, and synthetically generated data to simulate manipulation outcomes before acting^[7].

The hard requirement is temporal consistency over long horizons. A 10-second prediction at 30 FPS spans 300 frames that must hold object permanence, occlusion boundaries, and lighting, which pure pixel-space diffusion routinely breaks. Ha and Schmidhuber showed in 2018 that compressing dynamics into a VAE latent yields stable long-horizon rollouts, the principle now scaled to billion-parameter transformers^[8].

For world-model data, weight three things: multi-view coverage (4+ synchronized cameras per scene), physics diversity (rigid bodies, deformables, fluids, granular media), and labeled contact events (grasp start, collision, friction change). Every Truelabel delivery ships per-trajectory provenance and metadata, so a training sequence traces back to its raw capture when a model hallucinates implausible physics.

Teleoperation data: the highest-intent manipulation signal

Teleoperation datasets record a human solving a task through the robot, so the action distribution reflects expert strategy instead of a scripted heuristic. ALOHA collected 1,000 bimanual demonstrations of jobs like cable routing and reached 80%+ success after plain behavior cloning^[9]. DROID's 76,000 trajectories came through a force-feedback teleop rig that captures the contact-rich behavior scripted policies miss^[2].

Quality tracks two knobs. Interface fidelity: a 6-DOF SpaceMouse gives intuitive Cartesian control but no force feedback, while haptic exoskeletons like Franka's FR3 Duo let operators feel contact and tighten grasp precision^[10]. Temporal resolution: policies cloned from 10 Hz teleop move jerkier than 30 Hz ones because the transient dynamics get undersampled. For contact-rich work, 30 Hz is not optional.

Claru's warehouse dataset shows the density this needs: 2,400 pick-and-place sequences at 30 Hz with wrist-mounted RGB-D^[11]. When public coverage runs out, on-demand collection services take a client-specified task distribution, robot, and annotation schema^[12].

Sim-to-real transfer and domain randomization

Simulation-trained policies fail on hardware because of the reality gap: physics fidelity, sensor-noise models, and appearance all differ from silicon to the real world. Tobin et al. (2017) showed that randomizing lighting, textures, geometry, and dynamics in sim produces controllers robust to that shift^[13]. Peng et al. extended randomization to mass, friction, and actuator gains and hit 95% real-world success on locomotion trained purely in sim^[14].

Production pipelines pair huge sim diversity with a small real-world correction set. NVIDIA Isaac Gym runs millions of parallel randomized rollouts, then fine-tunes on 100 to 1,000 real trajectories to remove systematic bias. RLBench offers a 100-task sim benchmark for comparing transfer methods, but real hardware stays the only honest judge^[15]. Zhao et al. (2021) found policies claiming 90%+ transfer were often tested on cherry-picked scenarios^[16], so keep an independent validation set to catch it.

Multi-modal sensor fusion: RGB-D, LiDAR, tactile, proprioception

Physical AI fuses heterogeneous sensor streams, and each runs at a different rate and fails in a different way. PointNet processes LiDAR point clouds directly, without voxelization, for real-time 3D detection^[17]. The engineering cost is synchronization: a 5ms camera-to-LiDAR timestamp slip becomes a 15cm position error at highway speed, and a 2-degree extrinsic calibration error becomes a 10cm depth error at 3m. MCAP became the default container because it stores every stream with nanosecond timestamps and embedded calibration^[18].

Point-cloud labeling tools let annotators draw 3D boxes in the cloud while viewing the matching camera frame for context^[19]. Before you buy multi-modal data, confirm every stream shares one clock source and that extrinsics were validated against a known target.

Dataset formats: RLDS, MCAP, HDF5, and Parquet

Physical AI datasets ship in formats built for temporal, multi-modal data, and the format you accept sets your ingest cost. RLDS wraps TensorFlow Datasets with episode-trajectory semantics^[20]. MCAP is a self-describing container with microsecond timestamps, the ROS-bag replacement^[21]. HDF5 gives hierarchical, chunked storage but enforces no schema^[22]. LeRobot stores metadata as Parquet and video as MP4, 3 to 5 times smaller than raw frames with random access intact^[23].

Specify the target format in the contract. Converting a 500GB HDF5 set to RLDS burns real engineering days. Truelabel delivers in the buyer's format (RLDS, LeRobot, MCAP, or a custom schema) to S3, GCS, or Azure, so you skip the conversion.

Benchmark datasets: Open X-Embodiment, DROID, BridgeData V2, RoboNet

Public benchmarks are pretraining fuel, not deployment-ready data. Open X-Embodiment is the largest cross-embodiment corpus to date^[3]; DROID emphasizes real-world coverage and rejected 30% of raw collections for calibration errors or incomplete episodes^[2]. BridgeData V2 adds language-annotated kitchen demos for VLA training^[24], and RoboNet was the 2019 multi-robot pioneer whose 64x64 frames now limit its use^[25]. EPIC-KITCHENS-100 gives 100 hours of egocentric kitchen video with dense actions but no robot action labels^[26].

Scale trades against curation: Open X-Embodiment's size brings heterogeneous quality and mismatched annotation schemas. Treat any benchmark as a pretraining base, then commission the task-specific data (your pallet configs, box dimensions, lighting) that no generic set contains. Truelabel's research catalog profiles 750+ public and commercial physical-AI datasets with filterable metadata (robot platform, task, annotation type, license), so you can find the right pretraining source before paying for custom capture.

Licensing: CC-BY, CC-BY-NC, research-only, and custom terms

Physical AI licenses run from permissive to unusable. CC-BY 4.0 allows commercial use with attribution and covers BridgeData V2 and DROID^[27]. CC-BY-NC bans commercial use and is common in academic sets like EPIC-KITCHENS and Ego4D^[28]. Research-only terms, such as RoboNet's, forbid commercial deployment outright^[29].

The interpretation questions are not settled: does training a commercial model on CC-BY-NC data count as commercial use? Does deploying a model trained on research-only data breach the terms if you never ship the weights? GDPR Article 7 demands explicit consent for personal data, which bites when demonstrators or bystanders appear on camera^[30], and the EU AI Act requires dataset documentation for high-risk robotics^[31]. Truelabel's rights-cleared delivery includes contributor consent artifacts, location releases where applicable, and per-trajectory provenance, so the license posture is documented before you train.

1. 01

   Confirm commercial-use rights

   Separate research-only and non-commercial sets from anything you can deploy. One restrictive sub-dataset mixed into an aggregate contaminates the whole training corpus.

2. 02

   Check derivative and redistribution terms

   Some licenses permit training but forbid redistributing fine-tuned weights. Read the clause, not the SPDX tag.

3. 03

   Clear personal-data consent

   Any footage with identifiable people needs a consent basis under GDPR Article 7 and comparable regimes.

4. 04

   Verify embodiment and format fit

   Match joint count, gripper, workspace, and control frequency to your robot, and pin the delivery format (RLDS, MCAP, LeRobot) before purchase.

5. 05

   Demand provenance

   Require per-trajectory provenance and metadata so you can prove the data's lineage during a model audit.

Annotation requirements: 3D boxes, segmentation, keypoints

Physical AI needs spatially precise labels, and the precision, not raw volume, sets the labeling bill. 3D bounding boxes fix object pose and extent in world coordinates for grasp planning; semantic segmentation labels every pixel or point for navigation; keypoints mark grasp points and articulation joints. Polygon tools with interpolation cut annotator effort on tracking tasks^[32].

Precision scales with task tolerance: warehouse picking accepts ±2cm boxes, surgical robotics wants ±0.5mm keypoints. LiDAR cuboid labeling runs near 10cm position and 5-degree orientation accuracy^[33], while multi-modal suites track inter-annotator agreement across synced video and LiDAR^[34]. Tiered annotation services span plain boxes to dense segmentation, so match the tier to your tolerance rather than paying for surgical precision on a warehouse task^[35]. The dollar figures live in the cost breakdown below.

Cost structures: teleoperation, annotation, infrastructure

Physical AI data spends across three lines, and reading your own quote means knowing which one your spec loads. Teleoperation runs $50 to $200 per trajectory: a 2-minute pick-and-place at 30 Hz eats about 15 minutes of operator time with setup and review, or $75 to $150 at typical operator rates. Annotation runs $0.50 to $3.00 per 3D box, so a 10,000-frame driving sequence with 20 objects per frame lands between $100,000 and $600,000 for full 3D labels. Infrastructure amortizes: a 4-camera cell with lighting and compute costs $40,000 to $80,000, adding $4 to $8 per trajectory across 10,000 collections.

Standardized collection cells cut per-trajectory cost by 60% through scale^[36]. Model total cost of ownership before you build a rig: once you count pipeline engineering, internal collection often costs more than buying the spec. On Truelabel, post the spec and price the matched sample batch instead of a flat per-trajectory rate.

Data provenance and reproducibility

Most physical AI model failures trace back to data: a miscalibrated camera, dropped frames, mislabeled actions, an undocumented lighting change. Provenance systems record the full lineage from raw sensor log to training example, which is what makes root-cause analysis possible when a policy fails^[37]. OpenLineage gives a standard schema for tracking those transformations, already used in Airflow and dbt^[38].

Good provenance captures collection parameters (firmware, exposure, control frequency), processing steps (calibration, timestamp sync, outlier thresholds), and quality metrics (inter-annotator agreement, trajectory success, sensor dropout). Gebru et al.'s Datasheets for Datasets frames this as 57 questions across motivation, composition, collection, and preprocessing^[39]. The C2PA specification defines cryptographically signed, tamper-evident provenance for media files^[40]. For safety-critical work (autonomous vehicles, surgical robotics), require documented provenance to satisfy regulators, not as a nice-to-have.

Emerging trends: humanoids, dexterous hands, outdoor navigation

Humanoids are pulling demand toward bipedal locomotion and whole-body manipulation data. Figure AI's Brookfield partnership targets 1 million hours of humanoid teleoperation from warehouse deployments^[41], and NVIDIA's GR00T trains on a heterogeneous mixture of real-robot trajectories, human video, and synthetically generated data across locomotion, manipulation, and interaction^[7].

Dexterous manipulation stays data-starved: DexMV and HOI4D together offer 10,000 to 50,000 grasps, short of what a generalizable contact-rich policy needs. Kitchen-task datasets with tactile data (800 dexterous sequences) show the annotation density contact modeling demands^[42]. Outdoor navigation needs the seasonal, weather, and dynamic-obstacle coverage that indoor sets like RoboNet and BridgeData never captured.

NVIDIA's Physical AI Data Factory Blueprint packages simulation plus targeted real capture as a repeatable supply system rather than one-off annotation jobs^[43]. Expect humanoid and outdoor programs to move from research to deployment over the next two years, with sourcing, not model architecture, as the binding constraint.

Related pages

Use these to move from category-level context into specific task, dataset, format, and comparison detail.

External references and source context

1. RT-1: Robotics Transformer for Real-World Control at Scale

   RT-1 trained on 130,000 demonstrations across 700 tasks with 3 Hz RGB + 7-DOF actions

   arXiv ↩
2. DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

   DROID provides 76,000 trajectories from 564 scenes and 84 tasks

   arXiv ↩
3. Open X-Embodiment: Robotic Learning Datasets and RT-X Models

   Open X-Embodiment aggregates 60 datasets spanning over 1 million trajectories across robot platforms

   arXiv ↩
4. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

   RT-2 achieved 62% success on unseen tasks using PaLI-X 55B parameter backbone

   arXiv ↩
5. OpenVLA: An Open-Source Vision-Language-Action Model

   OpenVLA architecture treats actions as discrete tokens in autoregressive sequence

   arXiv ↩
6. NVIDIA Cosmos World Foundation Models

   NVIDIA Cosmos trains on 20 million hours of driving footage for video diffusion

   NVIDIA Developer ↩
7. NVIDIA GR00T N1 technical report

   GR00T N1 pretrains its world model on a heterogeneous mixture of real-robot trajectories, human videos, and synthetically generated data

   arXiv ↩
8. World Models

   Ha and Schmidhuber demonstrated VAE latent dynamics enable stable long-horizon rollouts

   worldmodels.github.io ↩
9. Teleoperation datasets are becoming the highest-intent physical AI content category

   ALOHA collected 1,000 bimanual demonstrations achieving 80%+ success after behavior cloning

   tonyzhaozh.github.io ↩
10. FR3 Duo

    Franka FR3 Duo bilateral teleoperation improves grasp precision by 40% via force feedback

    franka.de ↩
11. Teleoperation Warehouse Dataset for Robotics AI | Claru

    Claru teleoperation warehouse dataset provides 2,400 sequences at 30 Hz with RGB-D

    claru.ai ↩
12. Custom Robot Teleoperation Data Collection Service | Silicon Valley Robotics Center

    Silicon Valley Robotics Center offers on-demand teleoperation with custom task distributions

    roboticscenter.ai ↩
13. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World

    Tobin et al. showed randomized simulations produce controllers robust to real-world shift

    arXiv ↩
14. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization

    Peng et al. dynamics randomization achieved 95% real-world success on locomotion tasks

    arXiv ↩
15. RLBench: The Robot Learning Benchmark & Learning Environment

    RLBench provides 100-task simulation benchmark for sim-to-real transfer evaluation

    arXiv ↩
16. Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning

    Zhao et al. survey found policies claiming 90%+ sim-to-real often tested cherry-picked scenarios

    arXiv ↩
17. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

    PointNet processes LiDAR point clouds directly enabling real-time 3D object detection

    arXiv ↩
18. MCAP file format

    MCAP stores multi-modal streams with nanosecond timestamps and embedded calibration metadata

    mcap.dev ↩
19. segments.ai the 8 best point cloud labeling tools

    Segments.ai supports synchronized RGB-LiDAR annotation workflows for 3D bounding boxes

    segments.ai ↩
20. RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning

    RLDS wraps TensorFlow Datasets with episode-trajectory semantics and standardized schema

    arXiv ↩
21. MCAP specification

    MCAP is self-describing container format with microsecond timestamps for ROS bag replacement

    MCAP ↩
22. Introduction to HDF5

    HDF5 provides hierarchical storage with chunked compression for scientific datasets

    The HDF Group ↩
23. LeRobot dataset documentation

    LeRobot uses Parquet + MP4 achieving 3-5× compression versus raw image sequences

    Hugging Face ↩
24. BridgeData V2: A Dataset for Robot Learning at Scale

    BridgeData V2 offers 60,000 kitchen task demonstrations with language annotations

    arXiv ↩
25. RoboNet: Large-Scale Multi-Robot Learning

    RoboNet pioneered multi-robot datasets in 2019 with 15M frames from 7 platforms

    arXiv ↩
26. Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100

    EPIC-KITCHENS-100 provides 100 hours egocentric kitchen video with dense action annotations

    arXiv ↩
27. Attribution 4.0 International deed

    Creative Commons Attribution 4.0 permits commercial use with attribution

    Creative Commons ↩
28. Creative Commons Attribution-NonCommercial 4.0 International deed

    CC-BY-NC prohibits commercial use common in academic datasets

    creativecommons.org ↩
29. RoboNet dataset license

    RoboNet custom research-only license forbids commercial deployment

    GitHub raw content ↩
30. GDPR Article 7 — Conditions for consent

    GDPR Article 7 requires explicit consent for personal data use in datasets

    GDPR-Info.eu ↩
31. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence

    EU AI Act mandates dataset documentation for high-risk robotics applications

    EUR-Lex ↩
32. CVAT polygon annotation manual

    CVAT polygon annotation with interpolation reduces annotator effort by 60% for tracking

    docs.cvat.ai ↩
33. kognic.com platform

    Kognic provides LiDAR cuboid labeling with 10cm position and 5-degree orientation accuracy

    kognic.com ↩
34. encord.com annotate

    Encord supports synchronized video-LiDAR workflows with inter-annotator agreement dashboards

    encord.com ↩
35. scale.com physical ai

    Scale AI physical AI data engine offers tiered annotation from bounding boxes to segmentation

    scale.com ↩
36. scale.com scale ai universal robots physical ai

    Scale AI Universal Robots partnership deployed 100+ cells reducing per-trajectory costs 60%

    scale.com ↩
37. truelabel data provenance glossary

    Truelabel provenance system tracks training sequences contributing to failure modes

    truelabel.ai ↩
38. OpenLineage Object Model

    OpenLineage object model provides standardized schema for dataset transformation tracking

    OpenLineage ↩
39. Datasheets for Datasets

    Gebru et al. Datasheets framework proposes 57 questions for dataset documentation

    arXiv ↩
40. C2PA Technical Specification

    C2PA embeds cryptographically signed provenance metadata for tamper-evident audit trails

    C2PA ↩
41. Figure + Brookfield humanoid pretraining dataset partnership

    Figure AI Brookfield partnership targets 1M hours humanoid teleoperation data from warehouses

    figure.ai ↩
42. Kitchen Task Training Data for Robotics

    Claru kitchen task dataset includes 800 dexterous manipulation sequences with tactile data

    claru.ai ↩
43. NVIDIA: Physical AI Data Factory Blueprint

    NVIDIA Physical AI Data Factory Blueprint combines simulation with real-world collection for 10× efficiency

    investor.nvidia.com ↩

More glossary terms

FAQ