static masking

Static Masking is the original masking strategy used in BERT, where masking patterns were generated once during data preprocessing and fixed — meaning the model saw identical masked inputs for the same sentence across all training epochs.

The Process

• Preprocessing: Read corpus → Tokenize → Apply 15% masks → Save as TFRecords/HDF5.
• Training: Load saved records. Epoch 1 and Epoch 10 see identical `[MASK]` positions.
• Limitation: If training for 40 epochs, the model memorizes "Input A always has token 4 masked". It limits the variety of training signals.

Why It Matters

• Historical Context: Was an efficiency choice in original BERT (preprocessing is expensive).
• Obsolescence: Replaced by Dynamic Masking (RoBERTa) which proved superior.
• Lesson: Data augmentation (variation) usually beats static data in deep learning.

Static Masking is fixed-pattern training — a legacy approach where the training questions (masks) never changed, limiting the model's learning potential.