3D Motion Control in Video Generation: A New Approach

The generation of videos using artificial intelligence has made rapid progress in recent years. Controlling the movement of objects and people within generated videos is a particularly active field of research. While previous methods primarily relied on 2D control signals, the precise control of 3D movements opens up new possibilities for more realistic and complex videos.

3DTrajMaster: Controlling Multi-Entity Movements in 3D Space

A promising approach in this area is 3DTrajMaster. This method allows the manipulation of 3D movements of multiple entities (objects, people, etc.) within a generated video. In contrast to 2D methods, which can only control movement on the image plane, 3DTrajMaster uses 6DoF pose sequences (6 degrees of freedom: position and rotation in three dimensions). This allows the movement of each entity in 3D space to be precisely specified.

The core of 3DTrajMaster is a so-called "3D-Motion Grounded Object Injector". This combines the information about the individual entities with the specified 3D trajectories using a "Gated Self-Attention" mechanism. This architecture allows precise control of the entities' movement while preserving the relationships learned by the video generation model. This is crucial for the generalizability of the system, i.e., its ability to handle unknown scenarios and objects.

Challenges and Solutions

Generating high-quality videos with controlled 3D movement presents various challenges. To avoid degrading video quality, 3DTrajMaster uses a "Domain Adaptor" during training and an "Annealed Sampling" strategy during inference. The Domain Adaptor helps the model bridge the gap between the training data and the actual application cases. The Annealed Sampling strategy improves the quality of the generated videos by optimizing the sampling process during inference.

Another challenge is the lack of suitable training data. To address this problem, a special dataset, the "360-Motion Dataset," was created as part of the 3DTrajMaster project. This dataset contains 3D models of humans and animals animated with GPT-generated trajectories and then recorded from different viewpoints. The recordings were made with 12 evenly distributed cameras in various 3D environments created with the Unreal Engine.

Results and Outlook

Experimental results show that 3DTrajMaster delivers convincing results compared to previous methods, both in terms of motion control accuracy and generalizability. The precise control of 3D movements opens up new possibilities for video generation and editing and could find future applications in areas such as film, animation, virtual reality, and robotics.

Developments in the field of AI-powered video generation are dynamic and promising. Methods like 3DTrajMaster demonstrate the potential for creating increasingly realistic and complex videos that can be individually designed through precise motion control.

Bibliographie: https://openreview.net/forum?id=Gx04TnVjee https://github.com/KwaiVGI/3DTrajMaster https://openreview.net/pdf/939d81f065e4ddc37f9b13a334d71ab994ef69cf.pdf http://paperreading.club/page?id=271718 https://papers.cool/arxiv/2412.07759 https://arxiv-sanity-lite.com/ https://chatpaper.com/chatpaper/ja?id=4&date=1733846400&page=1 https://github.com/showlab/Awesome-Video-Diffusion https://paperreading.club/