Task data
Manipulation training data
Manipulation training data helps physical AI teams collect scoped examples in tabletop, shelf, bin, and drawer manipulation. When sourcing it, specify egocentric or wrist video with object and hand pose, target volume, delivery format, rights, consent, and QA rules for hands and manipulated objects in frame for active segments.
Quick facts
- Task
- Manipulation
- Modality
- egocentric or wrist video with object and hand pose
- Environment
- tabletop, shelf, bin, and drawer manipulation
- Volume
- a task-specific accepted-episode target
- Format
- LeRobot, RLDS, HDF5, or buyer-defined schema
- QA
- hands and manipulated objects in frame for active segments
Comparison
| Source | Use | Limitation |
|---|---|---|
| Public dataset | Research baseline | benchmark datasets often lack the buyer's object set and deployment environment |
| Internal capture | Maximum control | Slow setup and high fixed cost |
| truelabel sourcing | Spec-matched supplier response | Requires clear acceptance criteria |
What to specify for manipulation
The sourcing request should define task boundaries, capture setting, actor or robot requirements, accepted modalities, LeRobot, RLDS, HDF5, or buyer-defined schema delivery expectations, rights, consent, and what counts as an accepted sample. Registry sources show that task data is only reusable when collection setup and task distribution are explicit [1]. Buyers should also pin delivery expectations to formats and documentation they can validate before scale [2].
Why public data is usually not enough
benchmark datasets often lack the buyer's object set and deployment environment. Benchmark and vendor sources show that task labels, rights, and capture context are not interchangeable across deployments [3]. A buyer-specific request lets the team request the exact object set, environment, geography, and QA rubric needed for model training or evaluation.
Manipulation buyer scenario
A realistic manipulation request starts when a robotics team has a model behavior that fails in tabletop, shelf, bin, and drawer manipulation. The team does not just need more video; it needs examples where hands and manipulated objects in frame for active segments can be verified repeatedly [4].
[5]"CALVIN is a benchmark and toolkit for language-conditioned robot manipulation skills."
That means the supplier must show the requested egocentric or wrist video with object and hand pose, prove the capture context, and deliver LeRobot, RLDS, HDF5, or buyer-defined schema in a way the buyer can test before scaling.
Manipulation sample acceptance criteria
A useful sample for robot manipulation dataset should include at least one accepted episode, one borderline or failed example, a complete metadata manifest, and a note explaining how the supplier would scale only to a task-specific accepted-episode target [6]. If the sample cannot show hands and manipulated objects in frame for active segments, the buyer should reject it before funding a larger batch.
Manipulation task taxonomy and coverage
A robot manipulation dataset request should be task-specific, not template-level. Define the sub-tasks, capture viewpoints, object/environment coverage, labels, failure modes, and pilot acceptance rules before asking a supplier to scale. An accepted sample shows the hand or end-effector at contact, labels object state before and after, and keeps failed attempts with reasons.
| Planning area | Specify | QA question |
|---|---|---|
| Task phases | start/end boundaries, success, failure, recovery | Can reviewers identify every phase? |
| Sensors | wrist/external/egocentric video, state/action, depth/tactile if needed | Are streams synchronized and loadable? |
| Objects and environment | object family, layout, lighting, clutter, material | Does coverage match deployment? |
| Rights and provenance | license, consent, site permission, source manifest | Can legal/procurement audit the source? |
| Object state | before/after pose, open/closed, filled/empty | state change is implied but not labeled |
| Contact segment | approach, contact, manipulate, release | critical contact hidden by occlusion |
| Recovery | slip, retry, blocked path | all hard cases removed |
Manipulation accepted and rejected examples
Manipulation data is weak when it labels only the final success. Ask for approach/contact/release phases, object-state deltas, occlusion flags, and recovery attempts so the dataset teaches how manipulation unfolds, not just that it finished. Use licensing/provenance review for human or workplace footage, and route robotics-ready requests through the robot training data marketplace once the pilot schema is clear.
Manipulation pilot manifest fields
For Manipulation, the pilot manifest should include task phase, environment, object or route class, camera/sensor keys, action/state fields when applicable, timestamps, outcome label, failure reason, reviewer decision, rights/provenance files, and the target delivery format. An accepted sample shows the hand or end-effector at contact, labels object state before and after, and keeps failed attempts with reasons. The manifest is the bridge between supplier footage and buyer QA: if a reviewer cannot reproduce why a sample passed or failed, the dataset is not ready for scale-up.
Public datasets as references, not drop-in commercial supply
Public robotics datasets can guide schema and benchmark expectations, but commercial use, embodiment fit, action-state coverage, and consent/provenance are dataset-specific. Treat them as references unless official terms and buyer review support the intended use.
Pilot package before scale-up
Require a small loadable pilot with raw media/logs, manifest, labels, accepted and rejected samples, consent/provenance artifacts, and validation in the target format. Reject missing fields, broken sync, unclear boundaries, unsupported rights, and samples with only clean successes. For warehouse, kitchen, or industrial variants, compare the nearest warehouse, kitchen, or industrial sourcing spec before scaling.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- CALVIN GitHub repository
CALVIN is a benchmark and toolkit for language-conditioned robot manipulation skills.
GitHub ↩ - Project site
Meta-World is a simulated manipulation benchmark with many task variants.
meta-world.github.io ↩ - Project site
RLBench supplies language-conditioned robot manipulation benchmark tasks.
sites.google.com ↩ - Project site
Robomimic supports learning from demonstration datasets for robot manipulation policies.
robomimic.github.io ↩ - CALVIN paper
The CALVIN paper describes long-horizon manipulation skill evaluation for embodied agents.
arXiv ↩ - Project site
RoboCasa provides household manipulation task environments for robot learning evaluation.
robocasa.ai ↩ - truelabel egocentric data glossary
Internal contextual link to the egocentric data definition.
truelabel.ai - truelabel sourcing brief intake
Internal contextual link to Truelabel's sourcing intake workflow.
truelabel.ai - truelabel VLA training data sourcing
Internal contextual link to VLA training data sourcing.
truelabel.ai - truelabel warehouse robotics data sourcing
Internal contextual link to warehouse robotics data sourcing.
truelabel.ai - truelabel kitchen manipulation data sourcing
Internal contextual link to kitchen manipulation data sourcing.
truelabel.ai - truelabel LeRobot format guide
Internal contextual link to the LeRobot format guide.
truelabel.ai - truelabel LeRobot dataset alternative comparison
Internal contextual link to the LeRobot dataset alternative comparison.
truelabel.ai - truelabel eval data for robotics hub
Internal contextual link to robotics eval data sourcing.
truelabel.ai - truelabel teleoperation training-data page
Internal contextual link to teleoperation training data sourcing.
truelabel.ai - truelabel robot demonstrations training-data page
Internal contextual link to robot demonstration training data sourcing.
truelabel.ai - truelabel hand-object interaction data page
Internal contextual link to hand-object interaction training data requirements.
truelabel.ai - truelabel egocentric video datasets hub
Internal contextual link to the egocentric video datasets hub.
truelabel.ai
FAQ
What is robot manipulation dataset?
robot manipulation dataset refers to data collected for tabletop, shelf, bin, and drawer manipulation. It usually includes egocentric or wrist video with object and hand pose, metadata, and task outcomes that help train or evaluate physical AI systems.
What should a sourcing request include?
It should include task definition, environment, modality, volume, format, rights, consent, budget, deadline, and QA checks such as hands and manipulated objects in frame for active segments.
What format should buyers request?
LeRobot, RLDS, HDF5, or buyer-defined schema is the recommended starting point, but truelabel can route buyer-defined schemas when the training pipeline needs a custom layout.
Can this be exclusive?
Yes. Net-new sourcing requests can request exclusive commercial rights, while off-the-shelf datasets are usually non-exclusive unless the buyer explicitly purchases exclusivity.
What should a robot manipulation dataset request include?
Include target task phases, environment and object coverage, sensors/cameras, action/state fields when applicable, labels, success/failure outcomes, privacy/licensing artifacts, delivery format, and pilot acceptance criteria.
Sourcing data for robot manipulation dataset
Specify the environment, scale, and rights you need. Truelabel matches you with capture partners delivering robot manipulation dataset data with consent artifacts and commercial licensing attached.
Request manipulation training data