Evaluation of Language Models with S2S-Arena

The rapid development of large language models (LLMs) has greatly fueled interest in speech models, particularly Speech2Speech protocols that support speech input and output. However, existing benchmarks for evaluating the instruction-following capabilities of these models rely on automatic text-based evaluators and inadequately consider paralinguistic information in both speech understanding and generation. This concerns aspects such as intonation, emphasis, emotions, and speech pauses, which are crucial for natural and effective communication.

S2S-Arena: A New Approach to Evaluating Speech Models

To address this gap, S2S-Arena was developed, a novel arena-style benchmark that evaluates the instruction-following capabilities of speech models, taking into account paralinguistic information in realistic scenarios. S2S-Arena uses both synthesized speech (TTS) and live recordings to test the performance of the models in various contexts.

Structure and Methodology of S2S-Arena

The benchmark comprises 154 samples from four different areas: education, social interaction, entertainment, and medical advice. A total of 21 tasks were defined to test the models in different situations. The evaluation is performed manually in an arena style, i.e., by direct comparison of the performance of different models. Well-known language models such as GPT-4o-realtime, FunaudioLLM, and SpeechGPT were evaluated as part of the study.

Results and Insights of the Study

The results of the study provide important insights into the strengths and weaknesses of current language models:

First, it was shown that in addition to the superior performance of GPT-4o, cascaded models (ASR, LLM, TTS) perform better than jointly trained models after text-to-speech adaptation in Speech2Speech protocols. This suggests that the specialization of individual components offers advantages.

Second, the knowledge base of language models, considering paralinguistic information, mainly depends on the LLM backbone. However, multilingualism is limited by the speech module. This underlines the importance of the LLM as a central component.

Third, outstanding language models can already understand the paralinguistic information in the speech input quite well. However, generating audio with corresponding paralinguistic information remains a challenge. There is still considerable need for research in this area.

Outlook and Significance for the Future

S2S-Arena provides a valuable foundation for the further development of speech models. By considering paralinguistic information, the benchmark enables a more realistic evaluation of the performance and communication capabilities of speech models. The results of the study show that there is still room for improvement, particularly in the area of generating speech with paralinguistic nuances.

For companies like Mindverse, which specialize in the development of AI-powered speech solutions, these findings are of great importance. The development of chatbots, voicebots, and AI search engines benefits from a deeper understanding of the paralinguistic aspects of human communication. S2S-Arena contributes to advancing the development of even more powerful and natural language models.

Bibliography: https://arxiv.org/abs/2503.05085 https://arxiv.org/html/2503.05085v1 https://huggingface.co/papers https://github.com/FreedomIntelligence/S2S-Arena https://huggingface.co/FreedomIntelligence https://x.com/arxivsound?lang=de http://paperreading.club/category?cate=Speech http://paperreading.club/category?cate=Chat