```html Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition Multimodal large language models (MLLMs) are rapidly evolving, and expanding beyond individual domains is essential to meet the demand for more versatile and efficient AI systems. While previous models have primarily focused on text and images, Lyra integrates speech as a central component of its multimodal architecture. This article highlights Lyra's functionality, its advantages, and the research behind it. Lyra aims to enhance the multimodal capabilities of AI models, including the understanding of long speech sequences, the interpretation of sounds, the efficient processing of various modalities, and seamless speech interaction. Unlike many other omni-models, which incorporate speech only to a limited extent, Lyra places speech processing at its core. Three core strategies enable Lyra's efficiency and speech-centric capabilities: First, Lyra leverages existing open-source models and a novel multimodality LoRA (Low-Rank Adaptation). This approach significantly reduces training costs and the need for large datasets. LoRA allows for the adaptation of large language models to specific tasks without retraining the entire model. Second, Lyra employs a latent multimodality regularizer and extractor. These components strengthen the connection between speech and other modalities like images, thereby improving the model's overall performance. By jointly processing different modalities, Lyra learns to integrate the information from each source and develop a more comprehensive understanding. Third, Lyra is based on a large and high-quality dataset. This dataset comprises 1.5 million multimodal data samples (speech, image, audio) and 12,000 long speech samples. This enables Lyra to process complex, long speech sequences and achieve more robust omni-cognition. The size and diversity of the dataset contribute significantly to the model's performance and generalizability. Compared to other omni-methods, Lyra achieves state-of-the-art results in various benchmarks for image-language, image-speech, and speech-language. At the same time, Lyra requires fewer computational resources and training data. This efficiency makes Lyra a promising solution for a wide range of applications. The development of Lyra illustrates the trend towards increasingly powerful and efficient multimodal AI systems. The integration of speech as a central modality opens up new possibilities for interacting with AI and processing complex information. Future research could focus on expanding the supported modalities and improving interaction capabilities. Bibliography Zhong, Z., et al. "Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition." arXiv preprint arXiv:2412.09501 (2024). https://www.youtube.com/watch?v=7kh-M0jmmtI https://github.com/dvlab-research https://conf.papercept.net/conferences/conferences/SMC24/program/SMC24_ContentListWeb_4.html Marcus, A. (Ed.). (2014). Design, User Experience, and Usability: Theories, Methods, and Tools for Designing the User Experience: Third International Conference, DUXU 2014, Held as Part of the HCI International 2014, Heraklion, Crete, Greece, June 22-27, 2014, Proceedings, Part I (Vol. 8517). Springer. https://epoch.ai/data/epochdb/large_scale_ai_models.csv https://guoqiangwei.xyz/iclr2024_stats/iclr2024_submissions.html Lancaster, L., Lan, H., & Kenderdine, S. I. B. (2011). Omnidirectional 3D visualization for analysis of a large-scale corpus: The Tripitaka Koreana. In Culture and Computing. https://arxiv-sanity-lite.com/?rank=pid&pid=2402.14901 https://www.iimc.gov.in/files/downloads_documents/Communicator_July-Sept-2022.pdf https://core.ac.uk/download/548537788.pdf ```