Loss went down, quality went up, and the two barely talked to each other

A researcher trained a small text-to-image model, the same family of systems as Stable Diffusion or DALL·E, on a single GPU in 3.5 days. Three measurements came out of the run, and they contradict common assumptions about how these systems are trained and sampled.

Loss moved only from 0.805 to 0.754. Held-out image quality moved much further: a standard realism score (FID) fell from 33.7 to 27.0, a perceptual-distance score (FD-DINOv2) fell from 570 to 218, and detector-based object accuracy rose from 65% to 90%. Training and held-out loss stayed equal to the third decimal for all 24 epochs. The author's reading: flow-matching loss is a health signal, not a quality signal. Most of the loss at high noise is irreducible variance of the velocity target.

The run was small by design. A 210M-parameter cross-attention diffusion transformer trained on 4.2 million images at 256×256 on a single RTX PRO 6000. It sat on a frozen FLUX.2 autoencoder and a frozen flan-t5-base text encoder, so the model only had to learn how to connect captions to latent patches. Most of the time savings came from torch.compile, which delivered 2.4× over eager mode, plus a data pipeline that pre-bucketed images into five aspect ratios of around 256 tokens from step one.

The second finding is where the conditioning attention lands. The model has 16 register tokens, dedicated empty slots added to the image stream to absorb attention that would otherwise have nowhere productive to go, plus two learned key-value slots appended to every cross-attention block. At mid-noise in a middle block, those two slots received about 90% of the cross-attention mass. The EOS token, the usual "end of sequence" marker in language models, dropped to about 4%. Content words kept a few percent each, sharply on their objects. Register vectors also grew to 4 to 13 times the norm of ordinary image tokens by the middle blocks. The conditioning attention is not on the words. It lands on two learned null slots, and prior literature that pointed attention sinks at the EOS token was looking in the wrong place for setups like this one.

The third finding is a sampling-time knob that buys more than the obvious lever. A timestep shift, which rebalances how the sampler allocates steps across noise levels, is worth more than doubling the number of inference steps. On 2,456 held-out prompts with the final weights: 20 sampling steps with shift 2.8 → FID 27.0; 50 steps → 26.6; 8 steps → 28.4; 20 steps with no shift → 27.3 and FD-DINOv2 218 → 228. The shift value 2.8 comes from the SD3/RAE rule √(32·32·32/4096) for the 32-channel FLUX.2 latent. A 2.5× increase in steps with the shift moved FID by 0.4. A shift of 2.8 at 20 steps beat the no-shift baseline on the same compute budget. Most of the headroom at sampling time lives in the schedule, not the step count.

The limits matter as much as the findings. This is a single self-experiment at 256×256 with one specific stack: one researcher, one GPU, one run, frozen encoders, fixed seeds. The 90% attention figure is a single mid-noise, middle-block measurement, not a full layer and timestep sweep. The author did not claim, and these results do not show, that the same divergence would hold at 512² or 1024², at higher parameter counts, or with a different conditioning backbone. The recipe finding is the part that does generalize: an earlier recaptioned ~1M-image web crawl was worse than COCO alone on the same budget, so data quality dominated over scale at this regime.

The codebase, trained weights, and an in-browser demo are public, and the post's notes/ folder cites the source for each architectural decision. Three things are worth checking in the artifacts: whether the attention sink persists across all 16 blocks, whether loss and quality stay decoupled at higher resolution, and whether the shift-2.8 rule transfers to a different latent.