EgoLife: An AI Assistant for Everyday Life Through the User's Eyes

The development of AI assistants is progressing rapidly. A promising approach is the integration of this technology into wearable devices such as smart glasses. The EgoLife project pursues precisely this goal: the development of a personal AI assistant that optimizes and supports everyday life through the wearer's perspective.

To create the foundation for this assistant, the researchers conducted comprehensive data collection. Six participants lived together for a week and continuously documented their daily activities – from conversations and shopping to cooking and social interactions to leisure activities. The recording was done using AI glasses that captured multimodal, egocentric video data. In addition, synchronized video recordings were made from a third-person perspective to generate reference material.

The result of this study is the EgoLife dataset: a comprehensive, 300-hour dataset that includes multimodal and egocentric recordings of daily life from different perspectives and has been meticulously annotated.

EgoLifeQA: Answering Questions in the Context of Life

Based on this dataset, the researchers developed EgoLifeQA, a collection of tasks for answering questions in the context of daily life. These tasks aim to offer support to the wearer of the AI glasses in everyday situations. For example, the assistant can help remember past events, monitor health habits, and provide personalized recommendations.

EgoButler: The Combination of EgoGPT and EgoRAG

The development of such an AI assistant presents various challenges to research. These include the development of robust audiovisual models for egocentric data, reliable identification of people, and answering questions in a comprehensive temporal context. To address these challenges, EgoButler was developed – an integrated system consisting of EgoGPT and EgoRAG.

EgoGPT is a multimodal model trained on egocentric datasets and, according to the researchers, achieves state-of-the-art results in the field of egocentric video understanding. EgoRAG, on the other hand, is a retrieval-based component that enables answering questions in an extremely long context.

Experimental studies confirmed the functionality of both components and revealed critical factors and bottlenecks relevant for future improvements.

Outlook and Objectives

With the publication of the datasets, models, and benchmarks, the researchers aim to promote the further development of egocentric AI assistants. EgoLife offers a promising approach for integrating AI into everyday life and could become a valuable support in personal and professional environments in the future.

Bibliography: - https://github.com/EvolvingLMMs-Lab/EgoLife - https://x.com/NaveenManwani17/status/1896621586150801663 - https://egolife-ai.github.io/blog/ - https://huggingface.co/lmms-lab/EgoGPT-7b-EgoIT-EgoLife - https://twitter.com/liuziwei7 - https://x.com/choiszt/status/1896450152157925884 - https://huggingface.co/organizations/EgoLife-v1/activity/all - https://www.mmlab-ntu.com/conference/cvpr2025/index.html - https://proceedings.neurips.cc/paper_files/paper/2024/file/f5a8b5e5d007e66c929b971c2bc21d76-Paper-Conference.pdf