EgoCARBS: Egocentric Video Meets Active Acoustics for Kitchen Action Understanding

Egocentric video sees what is in front of you; a room camera sees all of you, but only in its room. We add the active acoustics of two everyday wearables, a smartwatch and camera glasses, and ask how far they take egocentric video toward the room camera's view of unscripted cooking.

Ego only Ego + wearablesEgo + exo

Two wearables, six streams, one clock

Smartwatchon the wrist
Samsung Galaxy Watch 7, Wear OS, unmodified: its own speaker, microphone (48 kHz) and IMU
Camera glasseson the face
zShades-2K camera glasses (46 g), with a small nRF52840 acoustic board (speaker, microphone, 50 kHz) on the frame
Room cameranot wornreference
A webcam on a tripod: the far end of the spectrum, and what the annotations are drawn on
Ego video

The sensing package at work

Held-out test moments, some from unseen users, people the models never saw in training. Often the room camera sees the person's back while the wearables follow the hands. The strip under each clip marks where the chosen setup names the verb right (green) or wrong (red).

Setup
Outcome
Users
Verb

The verbs in the dataset

The verb classes

Ego video (glasses)
Room camera
0.0 / 0.0 s

Predictions name the verb. Space plays or pauses, the arrow keys step a tenth of a second, and a click on any stream or row jumps there.