Huggingface100+
Hours
Long-context egocentric AR video
Full paper
Preview PDF
ECCV 2026In collaboration with
Ego
Everything
A Benchmark for Human Behavior–Inspired Long-Context Egocentric Video Understanding in AR Environment
Grounded in gaze traces, long-context egocentric video, and carefully
reviewed multiple-choice QA for
evaluating practical AR assistants.
Qiance Tang1†,Ziqi Wang1†,Jieyu Lin2,Ziyun Li2,
Barbara De Salvo2, and Sai Qian Zhang1‡
1New York University 2Meta Reality Labs
† Equal contribution. ‡ Corresponding author.
In collaboration with
EgoEverything
Full paperLong-context Egocentric Videos
Everyday egocentric video across people, places, and tasks.
02 / Dataset
INTRODUCE EGOEVERYTHING
EgoEverything is a benchmark for long-context egocentric video understanding in AR environments. It leverages human attention signals abstracted from gaze data to generate natural multiple-choice questions, covering over 5,000 QA pairs across more than 100 hours of video.
5,000+
MCQs
Multiple-choice memory and reasoning pairs
Real
Gaze Traces
Human attention signals aligned to video
8
Question Types
From recall to spatial-temporal reasoning
28
Object Categories
Fine-grained targets in everyday scenes
400+
Review Hours
Human validation and benchmark curation
Why EgoEverything
Existing long-context egocentric benchmarks often ask generic, template-like video questions and overlook how AR users naturally query what they attended to during interaction. EgoEverything fills this gap with gaze-grounded, human-reviewed MCQs that better reflect real AR memory and reasoning needs.
Question type distribution
What the benchmark asks
Percent share across EgoEverything MCQs.
Temporal-Spatial23.0%
Spatial-Spatial16.3%
Appearance14.6%
Item Presence13.7%
State Verify11.7%
Event Verify8.6%
Direct Location6.1%
Others5.9%
Target object coverage
What the questions are grounded in
Temporal-Spatial
Spatial-Spatial
Appearance
Item Presence
State Verify
Event Verify
Direct Location
Others
Question coverage
8 categories for AR memory
Temporal-Spatial
when + where
Spatial-Spatial
relative layout
Appearance
visual attribute
Item Presence
object exists
State Verify
state check
Event Verify
event happened
Direct Location
absolute place
Others
miscellaneous
How EgoEverything is built
EgoEverything turns long first-person video and gaze into compact, reviewed MCQs through a gaze-oriented target sampling pipeline.
01
Long egocentric video + gaze
Start from long AR recordings with gaze traces aligned to what the wearer attended to.
02
Gaze-oriented target sampling
Sample meaningful targets around attention patterns instead of asking generic video questions.
03
Reviewed MCQ
Turn sampled targets into multiple-choice questions and keep the benchmark human reviewed.
04 / Examples
MCQs are grounded in target objects, visual evidence, and reviewed answer choices
EgoEverything examples show how a gaze-sampled target object becomes a natural multiple-choice question with supporting frames, colored evidence, and model-versus-ground-truth labels.
Figure 4 / MCQ evidence examples
MCQ examples / 8 question types
01 / 08~60s clipTarget: striped blanketQ timestamp: 01:29
Was there a red and white chevron blanket on the sofa in the living room?
- Yes, it was on the armrest of the sofa.
- BNo, there was a plain red blanket on the sofa.
- CNo, an orange and blue striped blanket was on the sofa.
- DYes, but it was on the floor next to the sofa.
- EYes, but it was folded neatly on the cushion of the sofa.
Why Item Presence: Asks whether a specific item appears in a specified place.
Current VLMs remain well below human performance
The homepage preview surfaces the headline gap: strong models improve over text-only baselines, but still struggle with long-horizon visual memory, small details, and attention-conditioned evidence.
Leaderboard Snapshot
A 20.4 point gap from best VLM to human review.
Human83.5%
Reference performance from human review
Best VLM63.1%
Top evaluated vision-language model
Text-only35.9%
Question-only baseline without video
Accuracy
83.5%
Human
Accuracy
63.1%
Best VLM
Accuracy
35.9%
Text-only
05 / Analysis
Why current models struggle
EgoEverything exposes systematic limitations in current VLMs: longer recall intervals, targets farther from gaze, and smaller target objects.
01time
Longer recall interval
Performance drops as the answer depends on events farther back in the egocentric stream.
02gaze
Peripheral objects
Targets away from direct gaze are easier for humans to retain than for current VLM pipelines.
03scale
Smaller objects
Fine-grained, low-area visual evidence remains fragile even when the object is semantically simple.
06 / Citation
Cite the benchmark
arXiv BibTeX for the EgoEverything benchmark preprint.
BibTeX
@misc{tang2026egoeverythingbenchmarkhumanbehavior,
title = {EgoEverything: A Benchmark for Human Behavior Inspired Long Context Egocentric Video Understanding in AR Environment},
author = {Qiance Tang and Ziqi Wang and Jieyu Lin and Ziyun Li and Barbara De Salvo and Sai Qian Zhang},
year = {2026},
eprint = {2604.08342},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2604.08342}
}