ChartQAPro: A New Benchmark for More Complex Chart Question Answering

Interpreting charts is a crucial component of data analysis and decision-making. Automating this process through Chart Question Answering (CQA) systems promises more efficient and faster information retrieval. However, existing benchmarks like ChartQA reach their limits and do not reflect the complexity of real-world charts. With ChartQAPro, a new benchmark has been introduced that raises the challenges of chart question answering to a new level.

ChartQAPro comprises 1,341 charts from 157 different sources, including infographics and dashboards. The diversity of chart types and the inclusion of 1,948 questions of various kinds, such as multiple-choice, conversational questions, hypothetical questions, and questions without answer options, represent a significant improvement over previous benchmarks. This expansion of the dataset aims to better reflect real-world conditions in chart interpretation.

Initial tests with 21 different models show that ChartQAPro indeed exhibits higher complexity. For example, Claude Sonnet 3.5, a powerful Large Vision-Language Model (LVLM), achieves an accuracy of 90.5% on ChartQA, while performance drops to 55.81% on ChartQAPro. This significant decrease highlights the challenges ChartQAPro poses for current AI models.

The Importance of ChartQAPro for AI Development

The development of powerful CQA systems is of great importance for various application areas. From automated data analysis in companies to supporting researchers in evaluating complex data – the ability to correctly interpret charts and answer questions about them is essential. ChartQAPro makes an important contribution to the further development of AI models in this area.

By providing a more demanding benchmark, ChartQAPro enables a more accurate assessment of the capabilities of CQA systems. The identified weaknesses of current models provide valuable clues for future research and development work. The detailed error analyses and ablation studies conducted during the development of ChartQAPro offer important starting points for improving LVLMs in the field of chart understanding and interpretation.

Outlook

ChartQAPro represents an important step in the development of robust and reliable CQA systems. The benchmark allows researchers and developers to explore the limits of current AI models and work specifically on improving their abilities in the area of chart understanding. The availability of ChartQAPro via platforms like Hugging Face and GitHub facilitates access to this important dataset and promotes collaboration within the research community. The future development of CQA systems will benefit from the intensive use and further development of benchmarks like ChartQAPro and will contribute to advancing the automation of complex analytical tasks.