Mount a camera on the wall and it will see someone cooking. Mount it on the cook’s head and it sees what the hands are doing: which knife, which onion, how far through the cut.
That shift in viewpoint is why first person video AI training has become such a useful way to improve human activity recognition. Older recognition models learned from short third-person clips, then stumbled on the shaky, close-up footage that wearables produce.
Much of the recent progress came from a simple change. Teams began training on egocentric data from head-worn cameras, even before they had many labels.
Below, you’ll see what the research measured, where the method falls short, and six steps you can use on your own project.

What first person video AI training means
First person video AI training teaches a model to recognize actions from footage shot by the person doing them. The camera sits on a head strap, a pair of glasses, or a chest mount.
Human activity recognition, or HAR, is the job the model does with that footage, which means looking at video and naming the action, such as “wash cup” or “open drawer.”
A few terms come up again and again, so here they are in plain words:
- Egocentric and exocentric: Egocentric means first-person. Exocentric means third-person, like a security camera or a TV shot.
- Action recognition: The model labels a short clip that has already been cut around one action.
- Action detection: The model scans a long video and finds when each action starts and stops. This is harder.
- Pretraining: The model learns general visual patterns before it sees your task. Think of learning to read before you study law.
- Fine-tuning: The model then adapts to your task using labeled examples.
- Top-1 accuracy: This is how often the model’s first guess is right.
- mAP (mean average precision): This is a score for detection. It rewards a model for finding actions, timing them well, and naming them correctly. Higher is better.
Why the camera position changes what a model learns
Third-person models struggle with wearable footage because the two kinds of video look very different. Researchers at Meta AI listed these gaps when they explained why old transfer methods fall short.
| What differs | Third-person video | First-person video | Why it matters |
| Who is in frame | The whole person and the room | Mostly the wearer’s hands | The model must read actions from hands and objects |
| Video length | Short clips, often about 10 seconds | Long recordings, such as 8-minute Ego4D clips | The model needs memory of what came before |
| Label detail | Broad classes | Fine classes, such as “wipe kitchen counter” | Small differences change the answer |
| Everyday tasks | Chores are poorly covered | Cooking, cleaning, and play are common | Training data looks more like real use |
| Camera movement | Usually steadier | Sudden turns and motion blur | Frames are harder to read |
| Rare actions | Datasets are curated | Real life has many rare actions | Models must handle uneven data |
Here’s a simple example: a wall camera can tell that someone is standing at a sink, but their own body may block the view of what they are washing. A head camera shows the cup, the sponge, and the running water.
That said, the wall camera still wins for jobs that depend on posture or on the layout of the surrounding room. The Ego-Exo4D team describes the two views as partners: first-person shows hands and attention, while third-person shows the body and setting.
So a hospital that wants to spot falls should keep its room cameras, while a surgical training program that wants to review hand technique has a strong case for first-person video.
What the research shows about accuracy
The clearest result comes from Meta AI’s Ego-Only study, presented at ICCV 2023. A model pretrained on unlabeled first-person video beat the usual third-person baseline, and it needed 18 times fewer labels.

Source: Wang, Singh and Torresani, Ego-Only, ICCV 2023, Table 5. Scores are from the Ego4D Moments Queries validation set.
The table compares five ways to prepare the same detection model. The “ours” row is the Ego-Only method, which pretrained on unlabeled Ego4D video and then fine-tuned on Ego4D labels. It scored 16.3 mAP with about 14,000 labels.
The “exo-sup” row shows the usual approach, which pretrains on labeled third-person clips from Kinetics-400. It scored 13.9 and needed about 254,000 labels, or 18 times more.
The “scratch” row skips pretraining and uses the same 14,000 labels, yet it reached only 4.2, so the unlabeled footage clearly made the difference.
Third-person training can catch up, as the “exo-FT” row shows with a score of 16.2, but it costs far more labeling work. Unlabeled third-person video alone, shown in the “exo-MAE” row, stopped at 13.4.
Verbs improved more than nouns
On the EPIC-KITCHENS-100 recognition task, Ego-Only reached 73.3% top-1 accuracy on verbs, the best score in that comparison by 1.1 points. Its noun score of 59.4% trailed the top result of 63.9%.
The 2022 EPIC challenge report shows the same pattern from the other side. The winner pretrained on a huge private web video set and gained 7.0 points on nouns over the prior winner, but only 0.3 on verbs.
A useful rule follows from this: first-person footage teaches what hands are doing, while very large web datasets help a model name objects.
Narration and robots are benefiting too
Spoken or written notes can stretch a dataset further. The EgoVLP project built 3.8 million video and text pairs from Ego4D, then reported strong results on five first-person tasks.
The same footage now trains machines as well. NVIDIA’s EgoScale pretrained on 20,854 hours of human video and improved average robot success by 54% over a model with no pretraining. That measures robot skill, not recognition, but both depend on understanding hands and objects.
How first person video AI training works

Most projects move through five stages, and labeling comes late. That order keeps costs down, because the model learns a lot before anyone tags a single clip.
Pretraining is the stage that surprises newcomers, because the model teaches itself. In Ego-Only, it hid 90% of each video clip and tried to rebuild the missing parts, which forced it to learn how hands and objects tend to move.
The arrow at the end matters too. Each test shows which actions or people the model misses, and that tells you what to film next.
6 steps to apply first person video AI training
These steps follow one example: a warehouse that wants to spot packing errors. Swap in your own task as you read.
1. Write your action list before you film
Name each action as a verb and a noun. EPIC-KITCHENS uses this format, with 97 verbs and 300 nouns.
“Pack order” is too vague. “Scan item,” “fold carton,” and “seal box” give the model clear targets and give your labelers clear rules.
2. Start from a public dataset
You rarely need to begin from zero, so pick the dataset closest to your task and read its license before any commercial use.
| Dataset | Size | Good starting point for |
| Ego4D | 3,670 hours of daily life | Broad pretraining on everyday activity |
| Ego-Exo4D | 1,286 hours from 740 participants | Skilled work, sports, and paired views |
| EPIC-KITCHENS-100 | 100 hours and 90K actions | Fine hand and object actions |
| Charades-Ego | 8K videos and 157 classes | Indoor activity in paired views |
3. Match the training camera to the real camera
Film with the mount and lens you plan to use in production. A chest camera sees hands from a different angle than a head camera, so the footage won’t look the same to a model.
If your packers will wear chest cameras, collect at least some chest footage. Run a one-day pilot before you buy hardware in bulk.
4. Collect a lot of footage, then label a small part
Unlabeled video can be less expensive to scale than manually annotated footage, although collection, storage, and privacy requirements still add costs.
Growing that footage to 3,560 hours lifted the score to 15.5, so the first hours gave the biggest jump and later hours added less.
Label a small share first, then train and test. Add more labels only while your test score keeps rising.
5. Add sound and nearby moments
A wearable microphone sits close to the action, and that helps. The winning 2022 EPIC team noted that running water is a strong hint for “wash glass.” Adding audio and motion data raised its best single model from 52.7% to 53.6%.
Context helps as well. The runner-up gave its model clips from before and after each action, because “pour water in kettle” tends to sit between “open kettle” and “close kettle.”
In our warehouse, the tape gun has a distinct sound, so feed that audio to the model as well.
6. Test on new people and rare actions
Keep one whole site out of training and test there, since a random split can hide problems when the same workers and shelves appear on both sides.
Report three numbers: overall accuracy, accuracy on new people, and accuracy on rare actions. The 2022 EPIC winner scored 52.8% overall, 44.8% on unseen participants, and 28.5% on rare classes.
That drop is normal, and it also tells you where to collect your next batch of footage.
Where this helps in practice
First-person recognition fits work where hands tell the story. These are the most promising uses:
- Worker training: A system could flag a skipped step during packing, assembly, or lab work.
- Skills coaching: Ego-Exo4D includes tasks for estimating skill, with people ranging from novice to expert. A bike repair school could use that idea to give feedback on technique.
- Wearable assistants: Smart glasses could remember that you already added salt or took your pills.
- Robot teaching: A person could show a task once on camera, and a robot could use that footage as a starting point for learning.
Limits and mistakes to avoid
First-person video brings its own problems, and most of them show up after the pilot. Watch for these six:
- Ignoring privacy: Wearable cameras record coworkers, customers, screens, and homes. Ego4D uses consent and de-identification, and you should plan for both. Get legal advice before you film.
- Skipping fine-tuning: Unlabeled pretraining can’t do the whole job. In Ego-Only, frozen Ego4D features scored 7.8 mAP, while the fine-tuned model reached 16.3.
- Assuming more data always helps: Mixing Kinetics and Ego4D in pretraining scored 16.0, slightly below Ego4D alone. Off-topic footage can add cost without adding skill.
- Dropping third-person data too soon: It still helps with object names, posture, and room context. Keep it for the jobs it does well.
- Underestimating the footage: A 2026 survey in Machine Intelligence Research points to fast viewpoint shifts, motion blur, and blocked objects. It also says first-person datasets still lag third-person ones in scale and label quality.
- Trusting one benchmark: A method can shine on one task and barely move another. The EgoVLP team reported only a minor effect on one timing task, 0.616 versus 0.622.
Also, keep scores in context. A 16.3 mAP may look low, but that test asks a model to find and name many actions across long videos. Compare models on the same task and the same data.
Conclusion
First person video AI training improves human activity recognition by showing a model the hands, objects, and timing that a distant camera often misses. In the Ego-Only tests, it beat a common third-person baseline with 18 times fewer labels, and verb recognition saw the clearest gains.
You don’t need a giant project to try it, so pick one task, write a tight action list, pretrain on a public dataset, and label a small sample of your own footage.
Then test the model on people and places it has never seen, because that score will tell you more than any demo.
