data deduplication
Data deduplication is the process of identifying and removing duplicate or near-duplicate examples from a dataset. It is a critical data quality step for training language models, as duplicate data can waste compute, bias the model toward overrepresented content, and inflate evaluation metrics through train-test leakage.
Why Deduplication Matters
- Training Efficiency: Duplicate examples waste training compute on content the model has already seen.
- Memorization Risk: High duplication rates increase the chance of the model memorizing and regurgitating specific training examples verbatim.
- Evaluation Contamination: If duplicates exist across train and test splits, evaluation metrics are inflated.
- Distribution Skew: Overrepresented content biases the model toward certain topics, styles, or sources.
Deduplication Methods
- Exact Deduplication: Hash each example (using MD5, SHA-256) and remove exact matches. Fast and simple.
- URL Deduplication: For web data, deduplicate based on source URL before processing content.
- MinHash + LSH: MinHash creates compact signatures of document content, and Locality-Sensitive Hashing (LSH) efficiently groups similar documents. The standard approach for large-scale near-duplicate detection.
- Suffix Array: Build a suffix array over the concatenated corpus to find shared substrings. Used by the Llama and GPT training pipelines.
- Embedding-Based: Compute embeddings of each document and cluster by similarity. More expensive but catches semantic duplicates.
Scale Considerations
- Web-scale datasets like Common Crawl contain 30–50% duplicate content that must be removed.
- Efficient deduplication at trillion-token scale requires distributed, O(N) algorithms — exact comparison (O(N²)) is infeasible.
Best Practice: Apply deduplication at multiple granularities — document level, paragraph level, and even sentence level for critical datasets. The RefinedWeb dataset demonstrated that aggressive deduplication significantly improves downstream model performance.