Artificial Intelligence in the Service of Science: CURIE Tests the Limits of Large Language Models

The application of Artificial Intelligence (AI) in science promises revolutionary advancements. Large Language Models (LLMs) could support scientists in complex workflows and accelerate research. But how well are these models actually able to solve scientific problems? A new benchmark called CURIE (Context Understanding, Reasoning and Information Extraction) provides important insights.

CURIE: A Demanding Test Track for LLMs

CURIE was developed to evaluate the potential of LLMs in a scientific context. The benchmark comprises ten tasks with a total of 580 problems and solution pairs, curated by experts from six different disciplines. These include materials science, condensed matter physics, quantum computing, geodata analysis, biodiversity, and protein research. Both experimental and theoretical workflows are covered.

The tasks in CURIE place high demands on the LLMs. They require not only specialized knowledge in the respective disciplines, but also the understanding of extensive contextual information and multi-step reasoning. The models must synthesize information, recognize complex relationships, and ultimately generate solutions for scientific questions.

Evaluation Results: Light and Shadow

The evaluation of various LLMs with CURIE paints a mixed picture. While models like Gemini Flash 2.0 and Claude-3 demonstrate a consistently high level of understanding across various disciplines, others, like GPT-4 and command-R+, perform significantly worse on tasks related to protein sequencing. Overall, the best performance is at 32%, which makes it clear that there is still significant potential for improvement.

Outlook: CURIE as a Guide for Future Developments

The results of CURIE provide valuable insights into the strengths and weaknesses of current LLMs in a scientific context. They underscore the need to further develop models that possess both in-depth specialized knowledge and pronounced abilities in logical reasoning and information processing. CURIE can serve as a guide for future research and development efforts and contribute to unlocking the full potential of AI in science.

For Mindverse, a German company specializing in the development of AI-powered content solutions, the results of CURIE are particularly relevant. Mindverse offers an all-in-one platform for AI texts, images, and research, and develops customized solutions such as chatbots, voicebots, AI search engines, and knowledge systems. Understanding the limitations and possibilities of LLMs is essential for the further development of such systems.

Bibliography: - https://arxiv.org/abs/2503.13517 - https://openreview.net/forum?id=jw2fC6REUB - https://ml4physicalsciences.github.io/2024/files/NeurIPS_ML4PS_2024_240.pdf - https://arxiv.org/html/2503.13517v1 - https://github.com/google/curie/ - https://www.aimodels.fyi/papers/arxiv/curie-evaluating-llms-multitask-scientific-long-context - https://iclr.cc/virtual/2025/papers.html - https://openreview.net/revisions?id=1UCE6iaPHe - https://ml4physicalsciences.github.io/ - https://aclanthology.org/2024.naacl-long.205.pdf