Bitwise Autoregressive Modeling for High-Resolution Image Synthesis: A New Approach Called Infinity

The development of text-to-image models has made rapid progress in recent years. From pixel-based autoregressive models to diffusion models, various approaches have been explored to generate realistic and high-resolution images from text input. A new approach called Infinity now promises to push the boundaries of autoregressive models through bitwise modeling and an innovative tokenizer.

The Challenge of Scalability

Traditional autoregressive models struggle with scalability, especially when generating high-resolution images. Predicting each individual pixel or token becomes increasingly complex and computationally intensive as the resolution increases. This limits the application of these models for creating detailed images.

Infinity: A Bitwise Approach

Infinity bypasses these scaling problems by introducing a bitwise prediction framework. Instead of predicting entire pixels or tokens, the model focuses on individual bits. This allows for finer control over the generation process and permits the representation of subtle details. This approach is combined with an "infinite-vocabulary" tokenizer and classifier, which can theoretically scale the number of representable tokens infinitely. This significantly increases the expressiveness of the model compared to conventional autoregressive models.

Self-Correction Mechanism

Another important component of Infinity is the bitwise self-correction mechanism. This mechanism allows the model to detect and correct errors during the generation process. This improves the quality and consistency of the generated images.

Performance Compared to Diffusion Models

The developers of Infinity report impressive results. The model is said to outperform state-of-the-art diffusion models like SD3-Medium and SDXL in benchmarks like GenEval and ImageReward. Particularly noteworthy is the purported speed of Infinity: The generation of a 1024x1024 image is said to be possible in just 0.8 seconds, which is significantly faster than comparable diffusion models.

Potential and Future Research

Infinity represents a promising approach for high-resolution image synthesis. The combination of bitwise modeling, a scalable tokenizer, and a self-correction mechanism opens up new possibilities for generating detailed and complex images. The release of the models and code will allow the research community to further explore the potential of Infinity and utilize the technology for various applications in image generation and the modeling of unified tokenizers. Further research is necessary to explore the limits of this approach and evaluate its performance in various use cases. In particular, scaling the transformer size in conjunction with the infinite vocabulary size of the tokenizer could open up new challenges and opportunities in modeling.

Bibliography:

Chang, H., et al. "MaskGIT: Masked Generative Image Transformer." *CVPR*, 2022.
Ardizzone, L. "Generalization error bounds for kernel-based learning algorithms." *University of Heidelberg*, 2007.
Han, J., et al. "Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis." *arXiv preprint arXiv:2412.04431*, 2024.
Yu, J., et al. "Scaling Autoregressive Models for Content-Rich Text-to-Image Generation." *arXiv preprint arXiv:2206.10789*, 2022.
Tang, J., et al. "EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generation." *arXiv preprint arXiv:2409.18114*, 2024.
Ayan, B. K., et al. "BiGR: Harnessing Binary Latent Codes for Image Generation and Improved Visual Representation Capabilities." *arXiv preprint arXiv:2305.09149*, 2023.

Additional Online Resources:

r/ninjasaid13 (Reddit)
52CV/CVPR-2024-Papers (GitHub)
zhtjtcz/Mine-Arxiv (GitHub)
ECCV 2022 Papers