Egocentric video for robotics and physical AI
Physical AI is AI that acts in the real world, from a robot arm in a kitchen to an agent that works through a phone. Teaching it manual work takes footage of manual work. Egocentric video, recorded from the point of view of the person doing the task, is one of the few ways to show a model what a hand does, what the eyes see, and what comes next.
Why the worker's point of view matters
A camera across the room sees a person. A camera on the head or chest sees the task. It shows which object the hand reaches for, what is in view at that moment and how the view changes as the work goes on. That is the same view a robot camera has, which makes first-person footage closer to what an embodied model needs than footage from a fixed camera.
Written instructions leave most of this out. A recipe says "chop the onion". Video shows the grip, the angle of the knife, the slip and the correction. That is the detail a model learns from.
What models learn from it
| Model or task | What first-person video gives it |
|---|---|
| Vision-language-action (VLA) models | Camera views paired with an instruction, spoken or written, and the actions that followed. This is the pairing the model learns to produce. |
| Imitation learning | Many demonstrations of one task by different people, so a skill can be copied and generalized. |
| Hand-object interaction | How a hand approaches, grasps, holds, moves and releases an object. See how the labels work. |
| Computer vision | Objects, tools and scenes as a worker sees them, including hands that block the view and motion blur. |
| Planning and world models | The order of steps and what changes after each one, so a model can predict the result of an action. |
| Error recovery | Slips, spills and corrections. Scripted demonstrations leave these out, and a robot meets them all the time. |
The tasks labs ask for
Requests are usually for everyday manual work, filmed in real places by many different people. Variety of tasks, tools and settings matters more than the same task repeated.
- Food preparation and kitchen work
- Cleaning, sweeping and spill cleanup
- Sorting, packing and warehouse picking
- Assembly, repair and trade work
- Tidying and organizing a desk, a shelf or a room
- Software work on a screen, for agents that use a computer
What footage of people cannot do alone
A human hand is not a robot gripper, so first-person video is mostly used to teach a model how tasks are done, in what order and why, and to pretrain it before robot-specific data is added. A seller should not claim more than that. A buyer should say which part of the pipeline the footage is for.
What a buyer should say in a request
The tasks, the settings, how many hours and how many different people, the camera and the labels needed. The more specific the request, the faster a match comes back.
Read next
Questions about egocentric video for robotics and physical AI
What is physical AI?
AI that acts in the physical world, such as robots and other embodied systems, or agents that operate real tools. It needs examples of real actions, not only text.
Is first-person video enough to train a robot?
Usually not alone. It shows how people do tasks, which helps pretraining and planning. Robot-specific data is still needed for the final skill.
Do labs want one task or many?
Many, from many people. A set with variety teaches a model more than the same task repeated.
Why do labs care that hands are visible?
Hands carry the action. If they leave the frame or are hidden behind an object, the footage loses the part a model needs most.
Can screen recordings count?
Yes, for agents that use a computer. They show clicks, menus and corrections. Customer data on screen is masked first.
Didn't find your question? Ask us directly.