ibot

iBOT is a self-supervised vision transformer pre-training framework — performing masked image modeling (MIM) with an online tokenizer to learn high-level semantic abstractions without human annotations.

What Is iBOT?

• Definition: Image BERT Pre-training with Online Tokenizer.
• Core Mechanism: Distills knowledge from an online teacher network to a student network.
• Innovation: Avoids the need for a pre-trained tokenizer (unlike BEiT) by learning one jointly.
• Result: Learns both local (patch-level) and global (image-level) features simultaneously.

Why iBOT Matters

• Semantic Richness: Captures better semantic meaning than pure contrastive methods (like DINO).
• Efficiency: Eliminates the multi-stage training pipeline required by BEiT.
• Robustness: Performs exceptionally well on partial or corrupted images.
• Flexibility: Works on various vision transformer architectures (ViT, Swin).

Key Components

• Masked Image Modeling (MIM): Reconstructs masked patches (like BERT in NLP).
• Self-Distillation: Teacher network guides the student's learning.
• Online Tokenizer: Dynamically generates discrete tokens for image patches during training.

iBOT is a pivotal advance in self-supervised vision — bridging the gap between masked modeling and contrastive learning for superior visual understanding.