A New Approach for Photorealistic Avatars: Relightable Gaussian Codec Avatars

The creation of realistic digital avatars is a constantly growing field of research with applications in the film industry, video game development, and virtual reality. An important aspect is the realistic representation of light and shadow on the avatars, especially with different body postures and lighting conditions. A new approach, "Relightable Full-Body Gaussian Codec Avatars" (RFGCA), promises significant progress in this area.

Challenges in the Realistic Representation of Avatars

The realistic rendering of light and shadow on digital avatars, especially with complex movements and changing lighting conditions, presents developers with major challenges. Changes in body posture massively influence the light incidence on the body surface. Both local effects, such as the change in light incidence on individual body parts, and non-local effects, for example, due to the occlusion of body parts and the resulting cast shadows, must be taken into account.

The Solution: RFGCA

RFGCA addresses these challenges by cleverly decomposing the light transport calculation into local and non-local effects. For the local effects, so-called "Zonal Harmonics" are used. These mathematical functions allow for efficient calculation of diffuse light scattering and, in contrast to the frequently used spherical harmonics, can be efficiently rotated under joint movements. This allows the diffuse light scattering to be calculated in a local coordinate system, which decouples the calculation from the body posture.

Non-local effects, especially shadows, are modeled by a special neural network, the "Shadow Network". This network calculates shadows based on the pre-computed irradiance on a base model. This allows even complex cast shadows between different body parts to be realistically represented.

For the modeling of specular reflections and highlights, for example in the eyes, RFGCA uses a "Deferred Shading" method. This approach allows for a detailed representation of highlights and contributes to the overall quality of the avatar.

Advantages of RFGCA

The RFGCA approach offers several advantages over previous methods. The decomposition into local and non-local effects allows for an efficient and precise calculation of light scattering. The use of Zonal Harmonics simplifies the calculation under movement, and the Shadow Network ensures realistic cast shadows. The combination of these techniques leads to a convincing representation of avatars under different lighting conditions and body postures.

Applications and Future Prospects

RFGCA technology has the potential to revolutionize the creation of realistic avatars in various fields. In the film industry, photorealistic digital actors could be created that can be seamlessly integrated into real scenes. In video game development, players could immerse themselves in ever more immersive worlds, and in virtual reality, more realistic avatars could improve interaction and communication. Research in this area is progressing rapidly, and future developments could enable even more detailed and expressive avatars.

Bibliographie: - Saito, S., et al. "Relightable Gaussian Codec Avatars." CVPR 2024. - https://huggingface.co/papers/2501.14726 - https://arxiv.org/abs/2312.03704 - https://dl.acm.org/doi/10.1145/3680528.3687653 - https://openaccess.thecvf.com/content/CVPR2024/papers/Saito_Relightable_Gaussian_Codec_Avatars_CVPR_2024_paper.pdf - https://www.researchgate.net/publication/384213230_Relightable_Gaussian_Codec_Avatars - https://github.com/facebookresearch/goliath - https://arxiv.org/abs/2410.24223 - https://www.semanticscholar.org/paper/8719f022f6f0d67bb231d7c5317decf0e675e251 - https://dev.to/mikeyoung44/relightable-gaussian-codec-avatars-5fp8 - https://neurips.cc/virtual/2024/poster/97626