Optimization Theory and Practice: Surprising Agreements in Learning Rate Scheduling for Large Language Models

Training large language models is a complex task that requires significant computing power and sophisticated optimization strategies. A crucial factor for successful training is the choice of learning rate. A new study now shows surprising parallels between the theoretical foundation of convex optimization and the practical application of learning rate schedules when training large models.

Traditionally, learning rate schedules were developed heuristically and adjusted through empirical observations. The present research illuminates the connection to the theory of non-smooth convex optimization and provides a mathematical basis for the effectiveness of certain learning rate strategies. In particular, a connection is established between the performance limit from optimization theory and the behavior of learning rate schedules, such as the constant schedule with linear cooldown.

A remarkable result of the study is the explanation of the practical benefit of the cooldown phase. This phase, in which the learning rate is reduced towards the end of training, leads to improved convergence and generalization ability of the model. The theoretical analysis shows that the positive effect of the cooldown phase is reflected by the absence of logarithmic terms in the performance limit. This suggests that the cooldown phase is not only empirically advantageous but can also be theoretically justified.

The close agreement between theory and practice opens up new possibilities for optimizing learning rate scheduling. The researchers demonstrated this using Llama models with 124 million and 210 million parameters. By applying the insights from optimization theory, they were able to improve learning rate scheduling and achieve noticeable performance gains. Two concrete use cases were presented:

First, the learning rate schedule was extended for continued training with an optimal learning rate. This allows training to continue beyond the original timeframe while further improving convergence. Second, the optimal learning rate could be transferred between different learning rate schedules. This simplifies adaptation to new models and architectures, as the optimal learning rate does not have to be determined anew for each schedule.

The results of this study underscore the importance of a theoretical foundation for understanding and optimizing learning processes in large language models. The findings offer valuable starting points for the development of new and more efficient training strategies and contribute to exploiting the full potential of these models.

For companies like Mindverse, which specialize in the development and application of AI solutions, these research results are of particular interest. Optimizing learning rate scheduling is a central aspect in the development of customized AI solutions, such as chatbots, voicebots, AI search engines, and knowledge systems. By integrating these findings, training processes can be made more efficient and the performance of AI models can be improved.

Bibliographie: Agarwal, A., Kakade, S., & Wainwright, M. J. (2021). Optimal rates for stochastic convex optimization under Tsybakov noise condition. *Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing*, 143–154. Alvarez, F., Bolte, J., & Brahic, O. (2016). Hessian Riemannian gradient flows in convex optimization. *SIAM Journal on Control and Optimization*, *53*(2), 601–626. Bach, F., Jenatton, R., Mairal, J., & Obozinski, G. (2012). Optimization with sparsity-inducing penalties. *Foundations and Trends® in Machine Learning*, *4*(1), 1–106. Goodfellow, I., Bengio, Y., & Courville, A. (2016). *Deep learning*. MIT press. Schaipp, F., Hägele, A., Taylor, A., Simsekli, U., & Bach, F. (2025). The surprising agreement between convex optimization theory and learning-rate scheduling for large model training. *arXiv preprint arXiv:2501.18965*.