Whareformer: Learning to Track What is Where in Long Egocentric Videos

Jacob Chalk, Saptarshi Sinha, Dima Damen, Yannis Kalantidis, Diane Larlus

ECCV 2026

---

Whareformer is an online, long-term 3D object tracker for egocentric video that reasons jointly about where an object is and what it looks like, enabling long-term tracking which is robust through occlusions, out-of-frame periods, and evolving object appearance across EPIC-KITCHENS, IT3DEgo, and HD-EPIC.

Paper  •  Project Webpage  •  Architecture  •  Pre-extracted Features  •  Pre-trained Models

---

Table of Contents

• Citing
• Setup Environment
• Architecture
• Pre-extracted Features
• Pre-trained Model
• Training and Evaluating Whareformer
• License

---

Citing

If you find this code or our paper useful in your research, please consider citing:

@InProceedings{chalk2026whareformer,
    title     = {Whareformer: Learning to Track What is Where in Long Egocentric Videos},
    author    = {Chalk, Jacob and Sinha, Saptarshi and Damen, Dima and Kalantidis, Yannis and Larlus, Diane},
    booktitle = {European Conference on Computer Vision (ECCV)},
    year      = {2026}
}

---

Setup Environment

Option A - Latest packages

conda create -n whareformer python=3.13
conda activate whareformer
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu126
pip3 install pandas pyyaml pycolmap lmdb trimesh pyrender opencv-python decord scipy scikit-learn tqdm transformers xformers wandb 'git+https://github.com/cheind/py-motmetrics.git'
conda install conda-forge::open3d # If reconstructing EPIC meshes, otherwise not required
export PYTHONPATH=/path/to/Whareformer/src:$PYTHONPATH

Option B - Match our exact package versions (recommended for reproducibility)

conda create -n whareformer python=3.13.13
conda activate whareformer
conda install conda-forge::open3d=0.19.0 # If reconstructing EPIC meshes, otherwise not required
pip3 install -r requirements.txt

---

Architecture

We provide details on how the codebase generally operates in Architecture.

Pre-extracted Features

We provide pre-extracted features for all observations in EPIC, IT3DEgo, and HD-EPIC, along with the training data LMDB, here. Downloading these is recommended for reproducibility and ease of use.

If you wish to use the raw DINOv2 features without PCA applied, you can download them here.

Pre-trained Model

Our model weights are available here. Downloading these weights is recommended to reproduce the results reported in the paper.

---

Training and Evaluating Whareformer

Steps 0–3 are for training from scratch; you may need to update the relevant paths in the YAML config files. Skip to Step 4 if you are using our pre-extracted features and training data.

Step 0 - Download datasets

To use your own dataset instead, see Architecture for the expected output format before tracking.

Please refer to the original dataset pages for potentially more convenient links/methods to download the relevant parts of each dataset.

0a. EPIC-KITCHENS

Download EPIC-Kitchens RGB frames, VISOR Dense Interpolations, EPIC-Fields Sparse Reconstructions, and VISOR dense frame mapping.

The feature extractor assumes all data is extracted into one folder with the following file structure:

EPIC-Data
├── EPIC-Fields
│   ├── poses
│   │   ├── P01_01.json
│   │   ├── ...
│   │   └── P37_103.json
│   └── sparse
│       ├── P01_01
│       ├── ...
│       └── P37_103
├── EPIC-KITCHENS
│   ├── P01
│   │   └── rgb_frames
│   │       ├── P01_01
│   │       ├── ...
│   │       └── P01_104
│   ├── ...
│   └── P37
├── VISOR
│   ├── P01_01_interpolations.json
│   ├── ...
│   └── P37_103_interpolations.json
└── dense_visor_frame_mapping.json

0b. IT3DEgo

Download the dataset from the IT3DEgo repository.

The feature extractor assumes all data is extracted into one folder with the following file structure:

IT3DEgo
├── annotations
├── calibrations
├── enrollment_info
└── raw_videos

0c. HD-EPIC

Download videos, masks, and annotation information (mask_info and assoc_info) from HD-EPIC.

The feature extractor assumes all data is extracted into one folder with the following file structure:

HD-EPIC
├── videos
├── hd_epic_association_masks
├── frame_info.json
└── assoc_info.json

Step 1 - Extract features for each dataset

python scripts/extract_features.py --config config/epic/feature_extraction_config.yaml
python scripts/extract_features.py --config config/it3dego/feature_extraction_config.yaml
python scripts/extract_features.py --config config/hd_epic/feature_extraction_config.yaml

Note: EPIC feature extraction will reconstruct 3D meshes if not already present in the target scene_dir directory. This step takes a while, so we have pre-extracted them here.

Step 2 - Learn PCA on training data and apply to all datasets

python scripts/apply_pca.py \
    --output_dir /path/to/epic/features \
    --config_path config/epic/feature_extraction_config.yaml

python scripts/apply_pca.py \
    --output_dir /path/to/it3dego/features \
    --config_path config/it3dego/feature_extraction_config.yaml \
    --pca_path /path/to/pca_learned_on_epic

python scripts/apply_pca.py \
    --output_dir /path/to/hd_epic/features \
    --config_path config/hd_epic/feature_extraction_config.yaml \
    --pca_path /path/to/pca_learned_on_epic

Step 3 - Create training data

3a. Get oracle tracking data

python scripts/run_tracker.py --config config/epic/oracle_config.yaml

3b. Build LMDB database for training

python scripts/merge_lmdb.py --source_dir /path/to/oracle_training_data --split train

Step 4 - Train model

python training/train_model.py --config config/epic/whareformer_config.yaml

Step 5 - Run tracking inference

python scripts/run_tracker.py --config config/epic/whareformer_config.yaml

Step 6 - Evaluate results

python scripts/evaluate.py --results_dir /path/to/tracking_outputs

---

License

This project is released under the MIT License.