Evaluating Language Models Globally: GlotEval Enables Comprehensive Multilingual Testing

The rapid development of large language models (LLMs) is a global phenomenon. More and more regions are using these models for applications in their respective languages. However, evaluating these models in diverse language environments, especially for low-resource languages, poses a significant challenge for both academia and industry. Existing evaluation frameworks disproportionately focus on English and a few resource-rich languages. As a result, the actual performance of LLMs in multilingual and low-resource scenarios is often not adequately considered.

To address this gap, GlotEval was developed, a lightweight framework for massive multilingual evaluation. GlotEval supports seven core tasks: machine translation, text classification, summarization, open-ended generation, reading comprehension, sequence labeling, and intrinsic evaluation. These tasks cover dozens to hundreds of languages. The focus is on consistent multilingual benchmarking, language-specific prompt templates, and non-English-centric machine translation. This allows for a precise diagnosis of the strengths and weaknesses of models in different linguistic contexts.

A central concern of GlotEval is to provide a comprehensive and fair comparison of LLMs across different languages. The consideration of language-specific prompt templates is crucial here, as the performance of LLMs can depend heavily on the formulation of the input prompts. By adapting the prompts to the respective language, biases can be avoided and the comparability of the results improved.

Non-English-centric machine translation also plays an important role in GlotEval. Instead of considering translation exclusively from English to other languages, GlotEval allows for the evaluation of translations between different language pairs. This is particularly relevant for assessing the performance of LLMs in multilingual scenarios, where translations do not always involve English as an intermediary language.

A case study on multilingual translation demonstrates the applicability of GlotEval for multilingual and language-specific evaluations. The results show that GlotEval provides valuable insights into the performance of LLMs in different languages and can contribute to the development of more robust and versatile language models.

GlotEval addresses the growing need for comprehensive and nuanced evaluation of LLMs in an increasingly multilingual context. By focusing on consistent benchmarks, language-specific prompts, and non-English-centric translation, GlotEval offers a valuable tool for the development and deployment of LLMs in a variety of languages and applications.

The development of GlotEval underscores the importance of evaluation methods that reflect the linguistic diversity of the world. With the ongoing globalization of LLMs, the ability to accurately evaluate these models in different languages is becoming increasingly important. GlotEval makes a significant contribution to this development and enables a more informed assessment of the capabilities and limitations of LLMs in a multilingual context.

Bibliography: Luo, H., et al. "GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models." *arXiv preprint arXiv:2504.04155* (2025). Hu, Z., et al. "Open-Domain Question Answering with Retrieval-Augmented Generative Models." *arXiv preprint arXiv:2502.07346* (2025). Srivastava, A., et al. "GenBench: A Benchmark for Evaluating Generative Language Models." *Proceedings of the First Workshop on Generative Language Models*. 2024. Rönnqvist, S., et al. "MultiPragEval: Multilingual Pragmatic Evaluation of Large Language Models." *arXiv preprint arXiv:2307.03673* (2023). Polu, S., et al. "Can large language models reason about medical questions?" *arXiv preprint arXiv:2406.07736* (2024). Min, S., et al. "Towards Holistic Evaluation of Language Models." *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*. 2024. Su, H., et al. "Selective Question Answering under Partial Information." *Advances in Neural Information Processing Systems*, 32 (2019).