Randomized Autoregressive Visual Generation

Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, Liang-Chieh Chen
ByteDance
Paper arXiv Code

RAR is a an autoregressive (AR) image generator with full compatibility to language modeling. It introduces a randomness annealing strategy with permuted objective at no additional cost, which enhances the model's ability to learn bidirectional contexts while leaving the autoregressive framework intact. RAR sets a FID score 1.48, demonstrating state-of-the-art performance on ImageNet-256 benchmark and significantly outperforming prior AR image generators.


teaser image.


🔥 Highlights

1. We introduce RAR, an improved training strategy enabling standard autoregressive image generator to achieve state-of-the-art performance.


2. The proposed RAR is extremly simple yet effective: During training, we randomly permute the input token sequence with probability r, where r will starts at 1.0 and linearly decays to 0.0 over the course of training. This simple strategy enbales better bidirectional representation learning which is missing in standard raster-order-based AR image generator training.


3. RAR keeps the AR framework intact, and thus it is totally compatible to the LLM optimization techniques, such as KV-cache, leading to a significantly faster sampling speed compared to MAR-H or MaskBit while maintaining a better performance.



Abstract

This paper presents Randomized AutoRegressive modeling (RAR) for visual generation, which sets a new state-of-the-art performance on the image generation task while maintaining full compatibility with language modeling frameworks. The proposed RAR is simple: during a standard autoregressive training process with a next-token prediction objective, the input sequence—typically ordered in raster form—is randomly permuted into different factorization orders with a probability r, where r starts at 1 and linearly decays to 0 over the course of training. This annealing training strategy enables the model to learn to maximize the expected likelihood over all factorization orders and thus effectively improve the model's capability of modeling bidirectional contexts. Importantly, RAR preserves the integrity of the autoregressive modeling framework, ensuring full compatibility with language modeling while significantly improving performance in image generation. On the ImageNet-256 benchmark, RAR achieves an FID score of 1.48, not only surpassing prior state-of-the-art autoregressive image generators but also outperforming leading diffusion-based and masked transformer-based methods.

RAR Framework Overview

framework overview image.

Randomization with Annealing Leads to Substantial Perfromance Improvement

preliminary image.

RAR Works Well with Different Scan Orders

preliminary image.

Main Experiment Results

exp256 image. speed.

Scaling Behavior

scaling curve. scaling vis.

Visualization

vis1. vis2. vis3.

Visualization on the generated samples with RAR. Note that all images/videos in the project page are compressed, please refer to the paper for best views.

BibTeX

@article{yu2024randomized,
  author    = {Qihang Yu and Ju He and Xueqing Deng and Xiaohui Shen and Liang-Chieh Chen},
  title     = {Randomized Autoregressive Visual Generation},
  journal   = {arxiv},
  year      = {2024}
}