bit diffusion

Bit Diffusion is a diffusion model variant that represents discrete data as binary (bit) vectors and applies continuous diffusion in the binary representation space — encoding each discrete token as a set of bits, then treating each bit as a continuous variable for standard Gaussian diffusion.

Bit Diffusion Approach

• Binary Encoding: Convert discrete tokens to binary vectors — e.g., token ID 42 → [1,0,1,0,1,0,...].
• Analog Bits: Treat binary values as continuous — relax {0,1} to continuous values in [0,1] or ℝ.
• Gaussian Diffusion: Apply standard continuous diffusion to the analog bit vectors — add and remove Gaussian noise.
• Rounding: At generation time, round continuous values back to binary — decode to discrete tokens.

Why It Matters

• Best of Both: Combines the simplicity of continuous Gaussian diffusion with discrete output generation.
• Image Generation: Originally proposed for discrete image generation — pixel values as bit sequences.
• Scalability: Leverages the well-developed toolkit of continuous diffusion models for discrete problems.

Bit Diffusion is treating bits as continuous signals — encoding discrete data in binary and applying standard Gaussian diffusion for generation.