text shuffling

Text Shuffling describes a range of pre-training objectives involving the randomization of token or span order — forcing the model to rely on semantic coherence rather than just local syntax to reconstruct the original text.

Shuffling Levels

• Token Shuffling: Randomly shuffle tokens within a small window (e.g., 3-5 tokens) — de-correlates local position.
• Span Shuffling: Shuffle the order of spans or phrases.
• Sentence Shuffling: Permute full sentences (Sentence Permutation).
• N-gram Shuffling: Shuffle blocks of N-grams.

Why It Matters

• De-noising: Used in Denoising Autoencoders (DAE) and BART.
• Dependency Learning: If "President" and "Obama" are shuffled, the model must know they go together regardless of order.
• Regularization: Prevents the model from over-relying on strict sequential order (though in NLP order usually matters).

Text Shuffling is scrambling the message — forcing the model to reassemble order from chaos based on semantic relationships.