```html

Autonomous Experts: A New Approach for Mixture-of-Experts Models

Mixture-of-Experts (MoE) models have established themselves as a promising architecture in the world of machine learning. They are characterized by their ability to activate only parts of their parameters, often achieving higher efficiency and performance compared to dense models. A central component of this architecture is the so-called router, which distributes incoming data to specialized expert modules. However, the functionality of this router and its interaction with the expert modules are the subject of ongoing research and hold optimization potential.

The Challenge of Expert Selection

Traditional MoE models rely on a router that handles the assignment of input data (e.g., tokens in the context of natural language processing) to the individual expert modules. This router is usually based on a learned mechanism that analyzes the input and decides which expert is best suited to process the respective information. However, the separation between the router's decision-making and the actual execution of the computation by the experts can lead to suboptimal results. The router may not have enough information to accurately assess the actual capacity and current state of the experts. This can lead to experts being unevenly loaded or inputs being forwarded to experts that are not optimally suited for the respective task.

Autonomy of Experts: A New Paradigm Shift

A new approach that addresses the problem of expert selection is the concept of "Autonomy-of-Experts" (AoE). In contrast to traditional MoE models, where a central router controls the distribution of inputs, AoE allows the experts to independently select themselves for processing data. The basic idea behind this approach is that each expert knows best whether it is suitable for processing a particular input. This self-assessment is based on the analysis of the expert's internal activations. An expert that shows strong internal activations for a particular input signals its competence and its willingness to process this input.

Functionality of AoE Models

In AoE models, the router is completely eliminated. Instead, the experts first calculate internal activations for the incoming data. These activations are then compared, and only the experts with the highest activation norms are selected for further processing. The other experts remain inactive. To reduce the computational effort for the pre-calculation of the activations, techniques such as low-rank factorization of the weight matrices are used. This approach of self-assessment and subsequent comparison between the experts leads to improved expert selection and a more effective learning process.

Potential and Outlook

Initial results with pre-trained language models show that AoE models can achieve higher performance compared to traditional MoE models with comparable efficiency. The autonomy of the experts allows for a more dynamic and adaptive distribution of the computational load, which allows the potential of the MoE architecture to be better exploited. Future research will focus on further optimizing the AoE architecture and investigating its applicability to other areas of machine learning.

Bibliography: - https://huggingface.co/papers - http://arxiv.org/pdf/2308.11432 - https://openreview.net/pdf?id=BZ5a1r-kVsf - http://www.cs.toronto.edu/~fritz/absps/icann-99.pdf - https://openaccess.thecvf.com/content_CVPRW_2020/papers/w20/Pavlitskaya_Using_Mixture_of_Expert_Models_to_Gain_Insights_Into_Semantic_CVPRW_2020_paper.pdf - https://ieeexplore.ieee.org/document/9151017/ - https://www.researchgate.net/publication/382916607_THE_EVOLUTION_OF_MIXTURE_OF_EXPERTS_A_SURVEY_FROM_BASICS_TO_BREAKTHROUGHS - https://developer.nvidia.com/blog/applying-mixture-of-experts-in-llm-architectures/ - https://link.springer.com/content/pdf/10.1007/978-3-642-13480-7_7 - https://philsci-archive.pitt.edu/15530/1/Scientific%20Autonomy%20Draft16.pdf ```