Audiovisual Speech Recognition Revolutionized: Adaptive Models through Matryoshka LLMs

Audiovisual speech recognition (AVSR) uses both audio and visual information to improve the robustness of speech recognition, especially in noisy environments. The combination of both modalities allows the system to analyze lip movements and facial expressions and link this information with the acoustic signals. This is particularly helpful when audio quality is compromised by background noise. In recent years, large language models (LLMs) have demonstrated their capabilities in various areas of speech recognition, including AVSR.

However, integrating LLMs into AVSR systems presents a challenge due to the enormous amounts of data generated when processing speech representations. Directly processing this data through LLMs leads to significant computational overhead and high costs. Previous approaches attempt to solve this problem by compressing speech representations before inputting them into the LLM. However, higher compression rates often lead to performance degradation, requiring a trade-off between computational efficiency and recognition accuracy.

A New Approach: Matryoshka-based Multimodal LLMs

Matryoshka-based Multimodal LLMs offer an innovative solution to this problem. These models, inspired by the principle of Russian Matryoshka dolls, allow the simultaneous encoding of audiovisual representations at different granularity levels within a single model. This eliminates the need to train separate models for different compression levels. An example of this approach is Llama-MTSK, the first Matryoshka-based Multimodal LLM for AVSR. It allows flexible adaptation of the audiovisual token allocation to specific computational constraints without sacrificing performance.

For efficient fine-tuning of the LLM, various LoRA-based Matryoshka strategies have been developed that use global and scale-specific LoRA modules. LoRA (Low-Rank Adaptation) is a technique that allows training large language models with significantly reduced memory requirements. By applying LoRA in combination with the Matryoshka approach, the efficiency of the training can be further increased.

Evaluation and Results

Extensive tests on the largest AVSR datasets show that Llama-MTSK achieves excellent results and matches or even surpasses models trained independently with fixed compression levels. The flexible adaptability to different computational capacities makes Llama-MTSK a promising solution for use in real-world applications. The architecture enables efficient use of resources and scaling of performance as needed.

The development of Matryoshka-based Multimodal LLMs represents a significant advance in the field of AVSR. By combining flexible adaptability and high performance, new possibilities are opened for the development of robust and efficient speech recognition systems. Future research could focus on further optimizing the Matryoshka architecture and developing new training strategies to further improve the performance of AVSR systems in even more challenging environments.