Bernoulli Flow Models: Self-Consistent Generative Modeling for Binary Data
Hao Mo, Liying Yang, Shumin Yao, Xinxing Yu, Ajian Liu, Xudong Mao, Yanyan Liang
cs.LG, cs.CV
2026-10-08
BFM defines a closed-form Bernoulli posterior for any interval; 16 steps from a 256-step model hit FID 9.22 on LSUN Churches, and matched BLD falls to 204.10.
Binary variables take only 0 or 1 on each coordinate. Images can be packed into a binary latent, pixels can be thresholded into bits, and an Ising spin is already two-state. Analog Bits runs continuous diffusion on bit encodings. BLD diffuses directly in a binary latent. Samples are usable. Inference often still takes hundreds or thousands of network passes. NFE counts those passes, and one sampling step is usually one evaluation.
Continuous diffusion can cut that cost without a second training run. DDIM and DPM-Solver re-solve the same dynamics on a coarser time grid. Most binary diffusion models cannot. The forward process is a fixed one-step Markov chain, and the reverse posterior is built for that one step. Training ties the reverse update to this fine chain. Collapsing 256 steps to 16 at inference makes every update jump a long interval, while the sampler still inserts the one-step likelihood. Training and inference are no longer the same process.
The missing piece is a posterior that stays exact on an arbitrary interval.
Bernoulli Flow Models (BFM) specify the whole path from data to noise before choosing a grid. X0 is a data sample. The endpoint is binary white noise: each bit equals 1 with probability 0.5, independently. An intermediate Xt follows Bernoulli(t X0 + (1-t) X1), with t moving from 1 down to 0. Linear interpolation in a continuous model mixes coordinates. Here it mixes the probability that a bit is 1.
A fair-coin endpoint gives a closed-form marginal. The mix g(a, b) = a(1-b) + b(1-a) flips bit a with probability b and otherwise keeps it. Reaching time t is one flip of that kind, with marginal parameter α set by t. The forward move between neighboring times is the same flip. Its rate γ = 0.5(tprev - t) / tprev depends only on the endpoints. Bayes then yields the posterior. Given the current bit and the clean data, the previous bit is still Bernoulli, and its parameter is assembled from γ and α.
The network does x0-prediction: it predicts the clean bits under binary cross-entropy. The prediction is substituted into the posterior, and the previous state is sampled from that distribution. The implementation changes the path and the posterior inside the official BLD code. The backbone stays a full Transformer. Outputs are a sigmoid of the logits divided by a temperature κ, default 0.9. A smaller κ drives probabilities toward 0 or 1, so samples concentrate and diversity falls.
Theorem 1 applies the same posterior to any interval. If s is closer to the data than t, then γ = 0.5(s - t) / s and the formula is unchanged. Inference only splits the unit interval into K segments, and K can be far below the training length T. Fewer steps are new endpoints on the same path, not a one-step kernel forced across a multi-step jump. The paper calls the property self-consistency: a new grid does not add a structural mismatch between training and inference.
The closed form conditions on the true X0. Inference replaces X0 with a network prediction, and that error stays. What is removed is the approximation inside the transition kernel.
Backbone, optimizer, and schedule follow BLD. Adam uses β1=0.9 and β2=0.99, and the learning rate reaches 2×10⁻⁴ after 10K iterations. Training ran on two RTX 4090s with κ=0.9. FID, precision, and recall use 50K samples against the training set. LSUN and FFHQ are generated in a binary latent: BLD's pretrained autoencoder maps an image to 16×16×64 bits, and the decoder returns a 256×256 image.
The low-step test does not distill or retrain. On Churches, both models train for 256 steps with the linear schedule. Only the inference NFE changes, using the 100K-iteration checkpoints. At 16 steps BFM records FID 9.22 and BLD records 204.10. With a shared initial noise, BLD turns blurry as the budget shrinks. BFM keeps the scene outline.
On Bedrooms, from 100K to 400K iterations under the same architecture and optimizer, BFM keeps a lower FID. Point values along that curve are not listed.
The full-budget comparison uses 800K iterations and 64 steps. Public BLD checkpoints were not available, so BLD is a rerun from the official code in the same environment and configuration. StyleGAN and LDM figures are taken from earlier papers.
| Method | Steps | Bedrooms FID | Churches FID | FFHQ FID |
| BFM | 64 | 5.86 | 5.32 | 10.87 |
| BLD | 64 | 6.22 | 5.48 | 11.46 |
| D3PM | 200 | 6.60 | 6.02 | 9.49 |
| VQ-Diffusion | 200 | 7.19 | 6.88 | 8.79 |
| LDM-4/8/4 | 200 | 2.95 | 4.02 | 4.98 |
| StyleGAN | GAN | 2.35 | 3.86 | 4.16 |
BFM improves on BLD by 0.36, 0.16, and 0.59 FID. Precision is identical: 0.69, 0.66, and 0.68. Recall ties at 0.41 on Churches, 0.37 versus 0.36 on Bedrooms, and 0.49 versus 0.48 on FFHQ. StyleGAN and LDM remain ahead. On FFHQ, 10.87 is also worse than D3PM at 9.49 and VQ-Diffusion at 8.79.
Binarized MNIST sets every pixel above 0 to 1. BFM reaches FID 0.42, compared with CoVAE at 0.58, DMPM at 2.89, SFM at 4.62, and α-Flow at 5.02, under the SFM evaluation code. The sampling budget is not reported.
A scheduler ablation on Churches at 100K iterations covers linear, half-cosine, and reverse half-cosine paths. BFM scores 9.22, 9.41, and 9.27. BLD scores 9.55, 9.53, and 9.54. Linear wins, and the rest of the paper uses it. BLD near 9.5 shows that this table is not the 16-step protocol. The same 9.22 appears for linear BFM, and the paper does not state the table's NFE.
The short sampler is the part that can be used as is. A checkpoint trained for 256 steps is reused by redrawing the time grid, with no distillation and no consistency training. Sixteen forwards are one sixteenth of 256, and Churches FID stays at 9.22. BLD, crossing those gaps with a one-step posterior, falls to 204.10.
The full-step improvement is incremental. At 800K iterations and 64 steps, FID beats matched BLD by a few tenths. Someone who already has continuous diffusion or a GAN gains little by moving to a binary latent for that margin. The method fits tasks that already generate in bit space. The change is the path and the posterior.
Coarse-grid re-solving, routine for continuous diffusion, becomes a closed-form posterior on an arbitrary interval for Bernoulli variables, without an ODE solver on real numbers. Generic discrete diffusion and discrete flow matching do not supply that reparameterization.
Evaluation is limited to image generation and scientific binary benchmarks. The authors plan to add distillation or consistency training for very few steps and for one-step generation. The 16-step result supports training-free acceleration. It does not support one-step generation.
The 9.22 result is a 100K-iteration checkpoint, not the 800K model in the table. Low-NFE FID after full training is not reported. The Churches score of 5.32 uses 64 steps, not 16.
StyleGAN, LDM, D3PM, and VQ-Diffusion come from other training setups. The authors note that comparison with the originally published BLD numbers is not controlled, so the matched baseline is their rerun, BLD. Cross-paper FID is a scale check only.
The posterior is exact given the true X0. After a network replaces X0, closeness between short and full sampling rests mainly on one Churches curve and on images. Growth of prediction error with the interval is not measured. The Ising study sits in the appendix, with no numbers in the main text. On binarized MNIST the 0.42 versus 0.58 gap is small, and the step count is unknown. Temperature versus diversity is shown only as appendix samples.