Test-Time Preference Optimization: Dynamic Adaptation of Language Models through Iterative Feedback

Large language models (LLMs) impress with their capabilities but often struggle to adapt quickly to human preferences without retraining. A promising approach to solving this problem is called test-time preference optimization (TPO). TPO allows LLMs to be adapted to human preferences during the inference process, i.e., the generation of text, without changing the model parameters. This represents a significant difference from conventional methods that require retraining the model.

In contrast to approaches based on purely numerical reward signals, TPO translates these signals into textual critiques. These critiques then serve as the basis for iterative refinement of the model's responses. Through this process, the model learns to better understand and implement human preferences without the need for costly retraining.

Evaluations on various benchmarks, covering areas including instruction following, preference adaptation, safety, and mathematics, show that TPO progressively improves alignment with human preferences. Remarkably, the originally not instruction-tuned model Llama-3.1-70B-SFT, after only a few TPO steps, can outperform the already instruction-tuned Llama-3.1-70B-Instruct. This highlights the potential of TPO to significantly enhance the performance of LLMs in practice.

Another advantage of TPO lies in its efficient scalability in terms of both search breadth and depth during the inference process. This allows for flexible adaptation to different requirements and resources. Case studies have demonstrated how TPO leverages the inherent ability of LLMs to interpret and respond to reward signals. This ability is crucial for the effective implementation of preference optimization.

The results of previous research suggest that TPO represents a viable and resource-efficient alternative for optimizing preferences at test time. By dynamically adapting to human preferences during the inference process, TPO opens up new possibilities for the use of LLMs in various application areas. The ability to adapt the models "on-the-fly" simplifies integration into existing systems and reduces the effort required for model maintenance.

The combination of textual critiques and iterative refinement allows TPO to effectively leverage the strengths of LLMs while addressing their weaknesses regarding adapting to human preferences. Future research will focus on further exploring the potential of TPO and optimizing the method for even more complex application scenarios. The development of TPO represents an important step towards better human-machine interaction and could fundamentally change the way we interact with AI systems.

Bibliography: https://www.paperdigest.org/2024/06/icml-2024-highlights/ https://2024.emnlp.org/program/accepted_findings/ https://aclanthology.org/2024.findings-acl.424.pdf https://icml.cc/virtual/2024/events/2024SpotlightPosters https://www.chatpaper.com/chatpaper/zh-CN?id=3&date=1737561600&page=1 https://openreview.net/forum?id=cfn2O1qvxp https://arxiv.org/abs/2405.03803 https://iclr.cc/virtual/2024/events/spotlight-posters https://github.com/dair-ai/ML-Papers-of-the-Week https://nbogoychev.com/files/publications/2407.21783v2.pdf