Failures as Teachers: How Expert Errors Improve the Training of AI Agents

Large language models (LLMs) have demonstrated their potential as agents, especially in tasks that require multi-step reasoning processes and interactions. An established approach for fine-tuning LLMs as agents is Rejection Sampling Fine-Tuning (RFT). In this process, the LLM initially imitates successful action sequences (trajectories) provided by experts and subsequently improves its abilities through iterative fine-tuning based on self-generated, successful trajectories. However, since experts (e.g., GPT-4) are mainly successful in simpler subtasks, and RFT inherently favors easier scenarios, many complex subtasks remain unsolved and pose a persistent challenge (out-of-distribution, OOD).

A closer examination of these complex subtasks reveals that failed expert trajectories can often provide valuable clues, such as plans and key actions. This information can significantly improve the efficiency of the agent's exploration and the acquisition of critical skills.

Based on this observation, the "Exploring Expert Failures" (EEF) method was developed. EEF identifies useful actions from failed expert trajectories and integrates them into the training dataset. Potentially harmful actions are carefully excluded to avoid hindering the learning process. By leveraging the useful actions from expert failures, EEF manages to solve previously unsolvable subtasks and improve the performance of agent tuning.

Impressive Results and New Standards

The results of EEF are remarkable. In the WebShop environment, the approach achieved a success rate of 62%, surpassing RFT (53.6%) and GPT-4 (35.6%) and setting a new standard. To our knowledge, EEF is the first method to achieve a score above 0.81 in WebShop and above 81 in SciWorld.

These results highlight the potential of EEF to advance the development of more powerful LLM agents. By integrating information from failures, agents can handle more complex tasks and expand their capabilities more efficiently. The careful selection and integration of relevant information from the failed trajectories is crucial for the success of the method.

The further development of methods like EEF is essential to unlock the full potential of LLMs as agents and enable their use in complex applications. The ability to learn from mistakes is a central aspect of human intelligence, and transferring this principle to AI systems opens promising perspectives for the future of artificial intelligence.

Bibliography: Lan, L.-C., Bai, A., Cheng, M., Wang, R., Hsieh, C.-J., & Zhou, T. (2025). Exploring Expert Failures Improves LLM Agent Tuning. arXiv preprint arXiv:2504.13145. Huang, W., et al. (2024). Trial and Error Exploration-Based Trajectory Optimization of LLM Agents. ResearchGate. Li, Z., et al. (2024). Improving Large Language Model Agents with Tool Usage via Self-Supervised Pretraining and Instruction Tuning. Findings of the Association for Computational Linguistics: EMNLP 2024. Wu, S., et al. (2024). On the Role of Negations in Large Language Model Prompting: An Empirical Study of Negation-Aware Prompting Strategies. Findings of the Association for Computational Linguistics: EMNLP 2024. Sachin, T. (2024). Atlas: Approach to Finetune LLM Agents by Identifying Critical Steps in Expert Trajectories. Medium. "LLM Fine-tuning: Methods, Techniques, and Best Practices." SuperAnnotate, 2024.