Discover

'Unicode normalization' is a form of text normalization that transforms equivalent characters or sequences of characters into a consistent underlying representation so that they may be easily compared. Normalization is important when comparing text strings for searching and sorting (collation).

Contents

Composition and decomposition

Standards

See also

External links

Composition and decomposition

Underlying Unicode's normalization methods is the concept of character composition and decomposition. Character composition is the process of combining simpler characters into fewer precomposed characters, such as the n character and the combining ~ character into the single ñ character. Decomposition is the opposite process, breaking precomposed characters back into their component pieces.
Unicode composes combining characters and decomposes compatibility characters based on what it calls equivalence. Unicode has two standards of varying breadth for this equivalence: canonical, which distinguishes between functionally and what should be visually indistinguishable characters, and other compatibility characters, which may be visually and even semantically distinct. See the articles on Unicode equivalence and Unicode compatibility characters for more information.

Standards

Unicode defines four normalization standards.

'NFD' ''Normalization Form Canonical Decomposition''	Characters are decomposed by canonical equivalence.
'NFC' ''Normalization Form Canonical Composition''	Characters are decomposed and then recomposed by canonical equivalence. It is possible for the result to be a different sequence of characters than the original.
'NFKD' ''Normalization Form Compatibility Decomposition''	Characters are decomposed by compatibility equivalence.
'NFKC' ''Normalization Form Compatibility Composition''	Characters are decomposed by compatibility equivalence, then recomposed by canonical equivalence.

All the above methods will standardize the order in which decomposed characters appear, even sequences that were already decomposed prior to normalization. They may also replace characters or sequences with equivalent characters or sequences even if it doesn't result in the number of characters changing. These are done to achieve the consistency in encoding required for normalization.

See also

★ Unicode

★ Unicode equivalence

★ Unicode compatibility characters

★ Precomposed character

★ Ligature (typography)

External links

★ Unicode Standard Annex #15: Unicode Normalization Forms

★ utf8proc: A library for normalization of Unicode texts