Artificial Intelligence Enables Realistic Audio Generation for Long Videos

The synchronization of image and sound is essential for an immersive film experience. However, the automated generation of audio for long videos, such as films, presents a significant challenge due to the complexity of scene changes, dynamic semantic shifts, and the need for temporal precision. Existing video-audio synthesis methods, which are successfully applied to short videos, reach their limits with longer formats. Fragmented synthesis and a lack of consistency between scenes lead to an unsatisfactory result.

A new approach that has the potential to overcome these challenges is LVAS-Agent, a multi-agent framework. This system simulates the professional workflow of audio synchronization through the specialized collaboration of several AI agents. The audio generation process is divided into four steps: scene segmentation, script generation, sound design, and audio synthesis.

The innovation of LVAS-Agent lies in two central mechanisms. First, a discussion and correction mechanism that optimizes scene and script accuracy through exchange between the agents. Second, a generation and retrieval loop that ensures the semantic and temporal matching of image and sound. This loop allows the system to select the most suitable audio segments from a pool of generated segments, thus ensuring the coherence of the overall result.

To systematically evaluate the performance of LVAS-Agent and similar systems, LVAS-Bench was developed, the first benchmark of its kind. This benchmark comprises 207 professionally curated long videos covering a wide range of scenarios. Initial tests with LVAS-Agent on this benchmark show significantly improved audio-video synchronization compared to previous methods.

The development of LVAS-Agent and LVAS-Bench represents a significant advance in the field of video-audio synthesis. The ability to automatically generate realistic and synchronized audio tracks for long videos opens up new possibilities for the film industry, interactive media, and other applications. The improved temporal and semantic alignment of image and sound promises a significantly more immersive and coherent experience for viewers.

For companies like Mindverse, which specialize in AI-powered content creation, these advances open up new perspectives. Integrating systems like LVAS-Agent into existing platforms could significantly simplify and accelerate the automated production of high-quality video content. From generating marketing videos to developing interactive learning content, the possibilities are diverse.

Research in the field of video-audio synthesis is dynamic and promising. Future developments could include the integration of emotions and moods into the generated audio tracks to further enhance the narrative impact. Adaptation to specific genres and target audiences also represents an interesting field of research. The combination of AI-powered video and audio generation with other technologies, such as automated translation, could revolutionize the global distribution of video content.

Bibliography: Zhang, Y., Xu, X., Xu, X., Liu, L., & Chen, Y. (2025). Long-Video Audio Synthesis with Multi-Agent Collaboration. arXiv preprint arXiv:2503.10719. https://arxiv.org/abs/2503.10719
https://arxiv.org/html/2503.10719v2
https://openreview.net/forum?id=S7F7IMGX4O
https://paperswithcode.com/task/video-generation/codeless?page=15&q=
https://www.ijcai.org/proceedings/2023/0038.pdf
https://www.researchgate.net/publication/389748127_Automated_Movie_Generation_via_Multi-Agent_CoT_Planning
https://karine-h.github.io/GenMAC/
https://github.com/AGI-Edgerunners/LLM-Agents-Papers
https://github.com/showlab/Awesome-Video-Diffusion
https://paperswithcode.com/task/text-to-video-generation?page=6&q=