High-Resolution Image Processing: PS3 Enables Efficient Vision Pre-Training in 4K

The perception of fine visual details in high resolution is essential for many everyday tasks. However, current methods in vision pre-training are often limited to low resolutions (e.g., 378 x 378 pixels) due to the quadratically increasing computational cost with larger images. A new method called PS3 (Patch-based Selective Saliency) now promises to scale CLIP-like vision pre-training to 4K resolution without significantly increasing the computational cost.

Instead of performing contrastive learning on global image representations like conventional methods, PS3 focuses on the selective processing of local image regions. These are then matched with detailed, local image descriptions. This approach allows the learning of high-resolution representations with significantly reduced computational effort. The pre-trained PS3 model is capable of both encoding the entire image at low resolution and selectively processing local image regions at high resolution, based on their saliency or relevance to a text input.

By integrating PS3 into a multimodal language model (MLLM), a model called VILA-HD (Vision-Language-Model High-Definition) is created. Compared to baseline models without high-resolution vision pre-training, such as AnyRes and S^2, VILA-HD demonstrates significantly improved perception of high-resolution images. At the same time, VILA-HD requires up to 4.3 times fewer tokens. PS3 also provides VILA-HD with attractive scaling properties: The resolution can be increased without additional computational cost, and an increase in computational power at test time leads to improved performance.

Compared to current state-of-the-art models, VILA-HD outperforms existing MLLMs like NVILA and Qwen2-VL in various benchmarks and achieves higher efficiency than the latest token-pruning approaches. However, it appears that current benchmarks do not require 4K resolution for image perception, even though they contain 4K images. Therefore, 4KPro was developed, a new benchmark for image question answering (IQA) in 4K resolution. On this benchmark, VILA-HD surpasses all previous MLLMs, including a 14.5% improvement over GPT-4o and a 3.2% improvement as well as a 2.96x speedup over Qwen2-VL.

The development of PS3 and VILA-HD represents a significant advancement in the field of high-resolution image processing. By efficiently processing 4K images, new possibilities are opened for applications in various areas, such as medical imaging, satellite image analysis, and robotics.