script normalization

Script Normalization is the preprocessing step of unifying different Unicode representations of visually identical or semantically equivalent characters — ensuring that the model treats variations (e.g., Full-width vs. Half-width, composed vs. decomposed accents) as the same token.

Issues

• NFKC: Unicode Normalization Form KC is standard in NLP — converts "fi" (ligature) to "fi", full-width "Ａ" to "A".
• Diacritics: Removing accents (strip accents) vs. keeping them (café vs cafe).
• Zero-Width: Removing zero-width joiners/non-joiners common in Arabic/Indic scripts.

Why It Matters

• Token matching: Without normalization, "café" and "cafe" are totally different tokens to the model.
• Noise Reduction: Cleans up messy web text.
• Consistency: Essential for evaluating metrics (BLEU score) — don't penalize correct answers just because of an invisible Unicode difference.

Script Normalization is cleaning the text encoding — ensuring that the same character effectively always has the same digital byte representation.