pseudo-count methods

Pseudo-Count Methods are exploration techniques that extend count-based exploration to high-dimensional state spaces — using density models to estimate pseudo-counts hat(N)(s) that approximate traditional visit counts, enabling count-based exploration bonuses for complex observations like images.

Pseudo-Count from Density

• Density Model: Train a density model rho(s) on visited states.
• Pseudo-Count: hat(N)(s) = (rho(s)(1 - rho'(s)))/(rho'(s) - rho(s)) where rho' is the density after one additional visit.
• Bonus: r_bonus = beta / sqrt(hat(N)(s)) — same form as tabular count bonus.
• Models: PixelCNN, context tree switching, or other generative models for density estimation.

Why It Matters

• High-Dimensional: Extends count-based exploration to pixel observations — where tabular counts are infeasible.
• Theory Meets Practice: Bridges the theoretical elegance of count-based exploration with practical deep RL.
• Montezuma: Pseudo-counts enabled early progress on hard-exploration Atari games.

Pseudo-Count is counting in pixel space — using density models to approximate visit counts for scalable count-based exploration.