Character-Centric Movie Audio Descriptions through AI: FocusedAD

Audio description (AD) for films allows blind and visually impaired people access to visual content by describing the plot and events during dialogue-free scenes. In contrast to general video captioning, AD requires a precise, plot-relevant description with explicit mention of the characters involved, which presupposes a deep understanding of the film. A new approach called FocusedAD promises to improve this understanding using AI and revolutionize the creation of character-centric film descriptions.

The Challenges of Film Description

Creating audio descriptions for films presents particular challenges. It is not enough to simply describe the visible objects and actions. Rather, the AD must take into account the context of the scene, the emotions of the characters, and the development of the plot. The correct identification and naming of the acting characters is particularly important to ensure comprehensibility for the listeners. Previous approaches often struggled to meet these complex requirements.

FocusedAD: A New Approach to Audio Description

FocusedAD is an innovative framework specifically designed for the creation of character-centric film descriptions. It is based on three main modules:

The Character Perception Module (CPM) tracks the characters in the film and links them to their names. It analyzes the visual information and identifies the relevant characters in each scene.

The Dynamic Prior Module (DPM) integrates contextual information from previous ADs and subtitles. Learnable soft-prompts feed this information into the generation process, enabling a coherent and plot-related description.

The Focused Caption Module (FCM) finally generates the audio descriptions, which are rich in plot-relevant details and the names of the characters involved. By combining the information from the other modules, a comprehensive and understandable description of the film scene is created.

Automatic Creation of Character Databases

To address the challenge of character identification, FocusedAD also includes an automated pipeline for creating character databases. These databases contain information about the characters, such as their names and appearance, enabling efficient and reliable linking of visual information with the corresponding character names.

Convincing Results and Future Developments

Initial tests of FocusedAD show promising results. The framework achieves state-of-the-art performance on various benchmarks and also delivers convincing results in zero-shot learning, i.e., application to unknown data. Particularly noteworthy are the results on the MAD-eval-Named dataset and the newly developed Cinepile-AD dataset.

The development of FocusedAD is an important step towards an accessible film world. By combining state-of-the-art AI technology and a deep understanding of the requirements of audio description, FocusedAD enables an immersive film experience for blind and visually impaired people. Future research could focus on improving emotional contextualization and integrating further information sources to further enhance the quality of the generated descriptions.

Bibliography: Ye, X., Wang, C., Song, Y., Zhou, S., Li, L., & Bu, J. (2025). FocusedAD: Character-centric Movie Audio Description. arXiv preprint arXiv:2504.12157. Reviers, N. (2017). Audio Description Described: Current Standards, Future Innovations, Larger Implications. Journal of Specialised Translation, 35. Szarkowska, A. (2015). Audio Description Style and Film Experience: Description, Interpretation, Narration. Orero, P., & Wharton, S. (2007). ‘What Should I Say?’ Tentative Criteria to Prioritize Information in the Audio Description of Film Characters. MonTI. Monografías de Traducción e Interpretación, (1), 201-227. Fryer, L., & Freeman, J. (2021). WCAG 2.1 quick reference guide: A guide to understanding and implementing Web Content Accessibility Guidelines 2.1. W3C.