Egocentric video annotation for robotics: what the labels are
Raw footage shows what happened. Labels say what it means. Annotation turns a clip into training data by marking the steps, the hands, the objects and the outcome. A set with good labels is worth more to a buyer than the same footage without them, so it is worth knowing what each label is and who adds it.
The labels buyers ask for
| Label | What it is |
|---|---|
| Narration | A spoken or typed account of what the wearer is doing and why, kept in time with the video. |
| Action segments | The clip split into steps, each with a start time, an end time and a name, such as "pick up the cloth". |
| Hand trajectory | The path each hand takes through the scene over time, which shows how a movement is planned. |
| Hand pose and 3D hand mesh | The position of the wrist and fingers in each frame, and a 3D model of the hand, for fine manipulation. |
| Keypoints | Marked points on the hand, a tool or an object, such as the fingertips or the handle of a knife. |
| Object grounding | Linking a word in the narration to the object it names in the picture, with a box or an outline. |
| Hand-object interaction labels | Which hand touches which object, how it is held, and when the contact starts and ends. |
| Outcome and quality | Whether the task worked, and a reviewer note on how well it was done. |
Who adds the labels
Labels come from four places. The wearer can narrate while filming. The seller's own staff can label afterward. A labeling vendor can be hired. Or automated tools can propose labels that people check. Hand pose is often produced by software and corrected by a reviewer.
We do not label footage ourselves. We check which labels a seller already holds and say what is missing. Labeling after the fact is agreed in the contract, case by case, and the contract names who does it.
Quality checks that matter
- A written label guide, so the same thing is called the same name throughout
- More than one person labeling a sample, to see whether they agree
- A reviewer checking a share of the labels before the set is offered
- Labels kept in time with the video, with no drift between the two
- A clear record of which labels were made by hand and which by software
Labeling and privacy
People who label footage see it. Faces, voices and addresses should be removed first, and the contract should say who may see the raw files. See how de-identification works. A sample is cleaned and approved by the seller before a buyer sees it.
What to ask for or offer
A buyer should say which labels it needs and at what level of detail. A seller should list the labels it holds, the way they were made and a sample. A seller lists these in its summary, and a buyer states them in its request.
Read next
Questions about egocentric video annotation for robotics: what the labels are
What is egocentric video annotation?
Adding labels to first-person video: narration, steps, hand and object positions and outcomes, so a model can learn from it.
What is object grounding?
Linking a word or phrase, such as "the red cup", to the object in the picture. It teaches a model to connect language to what it sees.
Do I need hand pose labels?
Only if your model works on fine manipulation. Many buyers start with narration and action segments and add hand labels for a smaller set.
Can labels be added after the footage is recorded?
Yes, but it adds time and cost, and the contract has to say who does it and who sees the raw files.
Do labels change the price?
They can raise it, because the set is closer to ready to use. The amount is negotiated after we prepare the package, and no price is fixed in advance.
Didn't find your question? Ask us directly.