Multimodal Language Models Learn to Think Step-by-Step: R1-VL and StepGRPO

Artificial intelligence (AI) is developing rapidly, and large language models (LLMs) in particular have shown impressive progress in recent years. A key aspect of this development is the ability of these models to draw complex conclusions and think logically. A new approach, presented in a recent research paper, promises to significantly improve the reasoning ability of multimodal LLMs (MLLMs).

Traditionally, MLLMs are trained through supervised learning, by feeding them high-quality datasets containing examples of correct reasoning processes. This method, however, has a drawback: The models often only learn to imitate successful reasoning paths without truly understanding why certain steps are right or wrong. They fail to grasp the underlying logical principles and therefore may struggle in new, unfamiliar situations.

The new research paper proposes an innovative approach: Step-wise Group Relative Policy Optimization (StepGRPO). This framework is based on reinforcement learning and allows MLLMs to independently improve their reasoning skills by being rewarded or penalized for each individual step in the thinking process. In contrast to supervised learning, where the model merely copies successful examples, StepGRPO allows it to actively experiment and understand the consequences of its decisions.

At the heart of StepGRPO are two new rule-based reward mechanisms: Step-wise Reasoning Accuracy Reward (StepRAR) and Step-wise Reasoning Validity Reward (StepRVR). StepRAR rewards reasoning paths that contain necessary intermediate steps by performing a kind of "key step matching." StepRVR, on the other hand, rewards reasoning paths that follow a logically consistent structure by evaluating the completeness and logical validity of the thinking process.

By combining these two reward mechanisms, the model learns to both identify the correct steps and understand the logic behind the entire reasoning process. The result is R1-VL, a series of MLLMs that, according to the researchers, demonstrate outstanding capabilities in step-wise reasoning.

The effectiveness of StepGRPO and R1-VL was tested in extensive experiments on eight different benchmarks. The results show that the new approach significantly outperforms previous methods in terms of reasoning ability.

This development could have far-reaching implications for various AI applications. From medical diagnostics and scientific research to everyday tasks like planning trips, the ability to solve complex problems step-by-step and logically is crucial. Research on MLLMs and reinforcement learning methods like StepGRPO could pave the way for even more intelligent and powerful AI systems.

For Mindverse, a German company specializing in the development of AI solutions, these advancements are particularly relevant. The development of customized chatbots, voicebots, AI search engines, and knowledge systems directly benefits from the improvements in the reasoning ability of MLLMs. The new findings could contribute to developing even more powerful and intelligent AI solutions for businesses and customers.

Bibliography: https://arxiv.org/abs/2503.06749 https://arxiv.org/html/2503.06749v2 https://huggingface.co/papers?q=DeepSeek-R1 https://www.researchgate.net/publication/389786771_Towards_Reasoning_Era_A_Survey_of_Long_Chain-of-Thought_for_Reasoning_Large_Language_Models https://kargarisaac.medium.com/how-deepseek-r1-uses-reinforcement-learningto-supercharge-reasoning-3f826c2c8759 https://huggingface.co/papers/2501.12948 https://i-newcar.com/uploads/allimg/20250303/2-250303153331562.pdf https://www.youtube.com/watch?v=bAWV_yrqx4w https://www.facebook.com/groups/3670562573177653/ https://fetcher.alphaxiv.org/v2/pdf/2503.06749v2