Read, Allocate, Route: Role-Coupled Tokenization for Pixel-Space Generative Transformers
Jiongfan Yang, Weichen Fan, Haiwen Diao, Zhongang Cai, Ziwei Liu
ICLR 2027
cs.CV, cs.AI, cs.LG, cs.NE
2026-10-30
SpecSlot gives each token a fixed DCT band as its output role; only the lowest-frequency slot joins global attention. ImageNet-512: within 0.69 FID of JiT-B/16 at 38% lower cost.
Pixel-space generative Transformers (the JiT line) drop the separately trained autoencoder, but the full-resolution image must then become tokens. One patch grid decides three things at once: what the model reads, what each token predicts, and how many tokens enter global attention. On a 256×256 image, shrinking the patch from 16 to 8 takes the token count from 256 to 1,024 and multiplies the quadratic attention term by 16. Cascades and detailer heads move detail into extra components; window and sparse attention compress communication. None of them breaks the coupling between detail and the spatial grid.
The observation here is simple. In the frequency domain, low-frequency structure carries broad image organization and benefits from long-range exchange, while fine detail is mostly local. Whether a token should communicate globally should follow what it produces, not where it sits.
SpecSlot is a pixel-space rectified-flow Transformer whose training objective is identical to JiT's (the state, target, and loss stay in pixel space); all changes sit in tokenization and attention routing. One-patch-one-token becomes an explicit W×K allocation:
The design choices come with evidence. The prediction head is shared across bands, so the role is defined by the slot index, not per-band parameters. Slot 0 gets the carrier job because its signal is the first to cross DNR=1 in the flow, the first to emerge from noise. The spatial head is zero-initialized, so training starts from a pure spectral model. The structural payoff: W and K become independent knobs. The global quadratic term grows with W only, so raising K refines spectral detail without lengthening the global sequence.
Class-conditional ImageNet-256/512 and 256px text-to-image; main-table numbers are means over five independently trained seeds:
| Setting | Model | FID ↓ | Conditional GFLOPs ↓ |
| ImageNet-256 | SpecSlot-K4 | 6.59 | 129.2 |
| ImageNet-256 | JiT-CM (compute-matched control) | 6.68 | 124.2 |
| ImageNet-256 | JiT-B/8 | 4.34 | 218.5 |
| ImageNet-512 | SpecSlot-K4 | 9.14 | 135.6 |
| ImageNet-512 | JiT-CM | 11.98 | 141.2 |
| ImageNet-512 | JiT-B/16 | 8.45 | 219.6 |
At 256, SpecSlot-K4 ties JiT-CM (the gap sits inside the five-seed 95% interval) and descends faster at early-to-mid checkpoints; the quality ceiling still belongs to JiT-B/8 at 4.34, at roughly 1.7× the compute. At 512 the margin opens: 2.84 better than JiT-CM at 3.9% fewer FLOPs, and only 0.69 behind JiT-B/16 at 38.2% lower cost with 1.6× throughput. On text-to-image (BLIP3o plus JourneyDB data, frozen OpenCLIP ViT-L/14 on the text side), DPG-Bench 62.21 beats JiT-B/8's 61.15 at 40.6% lower forward cost; GenEval 26.99 still trails JiT-B/8's 29.78.
Ablations back the mechanism. Dual-basis output scores 6.59 versus 6.99 for the best single basis, so the spatial correction is not redundant. A compute-matched dense-global schedule lands at 6.81, worse than GGLL at the same price. Swapping the carrier to slot 1/2/3 raises FID to 6.91/7.21/6.97. Full global attention does reach 5.84, but at 74.8% higher cost; GGLL saves 42.8% of it.
The paper turns tokenization from a single patch-size choice into an explicit quality-compute allocation: detail scales with K while global communication does not move. For anyone training pixel-space models under an attention or compute budget, this is an actionable knob, and the code and Hugging Face weights are out. Be clear about the tier it buys: two nondominated operating points between JiT's low- and high-cost settings, not a new quality ceiling. The gain grows with resolution; at 512 each anchor covers four times the pixels of 256 while the global sequence stays at 288 tokens, which the paper offers as the explanation for the widening margin.
The compute-matched control JiT-CM is the authors' own construction (8×8 tokenizer, 128-dim input bottleneck, 32 context tokens at block 4, width 560), not a published model, so the headline comparison rests on a self-built baseline. The ImageNet-256 lead is 0.09 and inside the statistical interval; the real gap only appears at 512, and the paper concedes those results do not by themselves isolate the cause. K saturates: K4 to K8 buys 0.31 FID, the effective slot fraction drops from 0.803 at K2 to 0.495 at K8, and the authors state the more-anchors allocation rule only holds within the measured range. Text-to-image was tested at one resolution, and GenEval trails JiT-B/8 by about 2.8 points. All baselines are pixel-space JiT variants; there is no head-to-head against latent-diffusion models.