Improved Evaluation of Language Models with EvalTree: Precisely Identifying Weaknesses

Evaluating language models (LMs) is a crucial step in their development and improvement. An ideal evaluation method should not only uncover weaknesses but also provide concrete guidance for optimization. A new approach called EvalTree pursues precisely this goal by creating a detailed profile of an LM's weaknesses.

EvalTree goes beyond mere performance evaluation and generates a list of weaknesses formulated in natural language. This list is based on the LM's performance on each individual task of a benchmark. To compare the effectiveness of different methods for weakness profiling, quantitative evaluation metrics have been developed.

At the heart of EvalTree is the construction of a so-called Capability Tree. Each node of this tree represents a specific capability, described in natural language, and is linked to a subset of the benchmark instances that specifically test this capability. By analyzing the nodes where the LM performs poorly, EvalTree creates a detailed weakness profile.

Tests with benchmarks such as MATH and WildChat show that EvalTree identifies weaknesses more precisely and comprehensively compared to other methods. This precise identification enables targeted data collection and augmentation. Data collection based on EvalTree has proven to be more effective than other strategies for improving LM performance.

Furthermore, EvalTree can also be used to uncover weaknesses in existing evaluation practices. For example, it was able to highlight shortcomings in the human-based evaluation of Chatbot Arena.

To promote further research and application, the developers of EvalTree have released their code and an interactive user interface. With this interface, users can explore the Capability Trees created by EvalTree and gain a deeper understanding of the strengths and weaknesses of the examined LMs.

The development of EvalTree represents a significant advancement in the evaluation and improvement of language models. The method enables targeted optimization based on the specific weaknesses of the respective LM, thus contributing to more efficient further development. By providing code and an interactive interface, further research and application of EvalTree is actively encouraged.

Bibliography: - Zeng, Z., Wang, Y., Hajishirzi, H., & Koh, P. W. (2025). EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Trees. arXiv preprint arXiv:2503.08893. - https://www.themoonlight.io/review/evaltree-profiling-language-model-weaknesses-via-hierarchical-capability-trees - https://x.com/gm8xx8/status/1902192404226085251 - http://paperreading.club/page?id=291582 - https://zhiyuan-zeng.github.io/ - https://www.researchgate.net/scientific-contributions/Yizhong-Wang-2131086534 - https://koh.pw/ - https://www.researchgate.net/scientific-contributions/Hannaneh-Hajishirzi-2047656727 - https://x.com/zhiyuanzeng_?lang=de