Character Model for the World Wide Web 1.0: Normalization

3.1.1 Why do we need character normalization?

Text in computers can be encoded in one of many character encodings. In addition, some character encodings allow multiple representations for the 'same' string, and Web languages have escape mechanisms that introduce even more equivalent representations. For instance, in ISO 8859-1 the letter 'ç' can only be represented as the single character E7 'ç', but in a Unicode encoding it can be represented as the single character U+00E7 'ç' or the sequence U+0063 'c' U+0327 '¸'. In HTML it could be additionally represented as ç or ç or ç (five equivalent representations in total).

There are a number of fundamental operations that are sensitive to these multiple representations: string matching, indexing, searching, sorting, regular expression matching, selection, etc. In particular, the proper functioning of the Web (and of much other software) depends to a large extent on string matching. Examples of string matching abound: parsing element and attribute names in Web documents, matching CSS selectors to the nodes in a document, matching font names in a style sheet to the names known to the operating system, matching URI pieces to the resources in a server, matching strings embedded in an ECMAScript program to strings typed in by a Web form user, matching the parts of an XPath expression (element names, attribute names and values, content, etc.) to what is found in an XML instance, etc.

String matching is usually taken for granted and performed by comparing two strings byte for byte, but the existence on the Web of multiple character representations means that it is actually non-trivial. Binary comparison does not work if the strings are not in the same character encoding (e.g. an EBCDIC style sheet being directly applied to an ASCII document, or a font specification in a Shift_JIS style sheet directly used on a system that maintains font names in UTF-16) or if they are in the same character encoding but show variations allowed for the 'same' string by the use of combining characters or by the constructs of Web languages.

Incorrect string matching can have far reaching consequences, including the creation of security holes. Consider a contract, encoded in XML, for buying goods: each item sold is described in a Stück element; unfortunately, "Stück" is subject to different representations in the character encoding of the contract. Suppose that the contract is viewed and signed by means of a user agent that looks for Stück elements, extracts them (matching on the element name), presents them to the user and adds up their prices. If different instances of the Stück element happen to be represented differently in a particular contract, then the buyer and seller may see (and sign) different contracts if their respective user agents perform string identity matching differently, which is fairly likely in the absence of a well-defined specification for string matching. The absence of a well-defined specification would also mean that there would be no way to resolve the ensuing contractual dispute.

Solving the string matching problem involves normalization, which in a nutshell means bringing the two strings to be compared to a common, canonical encoding prior to performing binary matching. (For additional steps involved in string matching see 4 String Identity Matching.)

There are options in the exact way normalization can be used to achieve correct behavior of normalization-sensitive operations such as string matching. These options lie along two axes: i) when normalization is performed, and ii) what canonical encoding is used. The next subsections discuss these axes.

3.1.2 Early or late normalization

The first axis is a choice of when normalization occurs: early (when strings are created) or late (when strings are compared). The former amounts to establishing a canonical encoding for all data that is transmitted or stored, so that it doesn't need any normalization later, before being used. The latter is the equivalent of mandating 'smart' compare functions, which will take care of any encoding differences.

There are several advantages to early normalization, as follows:

• Almost all legacy data as well as data created by current software is normalized (if using NFC).

• The number of Web components that generate or transform text is considerably smaller than the number of components that receive text and need to perform matching or other processes requiring normalized text.

• Current receiving components (browsers, XML parsers, etc.) implicitly assume early normalization by not performing or verifying normalization themselves. This is a vast legacy.

• Web components that generate and process text are in a much better position to do normalization than other components; in particular, they may be aware that they deal with a restricted repertoire only, which simplifies the process of normalization.

• Not all components of the Web that implement functions such as string matching can reasonably be expected to do normalization. This, in particular, applies to very small components and components in the lower layers of the architecture.

• Forward-compatibility issues can be dealt with more easily: less software needs to be updated, namely only the software that generates newly introduced characters.

• It is a prerequisite for comparison of encrypted strings (see [CharReq], section 2.7).

Early normalization also has downsides: everyone must play by the same rules, and things break down when a producer of text data doesn't play by the rules. Furthermore, the location of the error (typically at a recipient that assumes proper normalization) is remote from the source (the faulty producer).

When recipients cannot count on early normalization, then some form of late normalization is the only way to ensure proper results of string comparison and other normalization-sensitive operations.

3.1.3 The choice of Normalization Form

The second axis is a choice of canonical encoding. This choice needs only be made if early normalization is chosen. With late normalization, the canonical encoding would be an internal matter of the smart compare function, which doesn't need any wide agreement or standardization.

By choosing a single canonical encoding, it is ensured that normalization is uniform throughout the web. Hence the two axes lead us to the name 'early uniform normalization'.

The Unicode Consortium provides four standard normalization forms (see Unicode Normalization Forms [UTR #15]). These forms differ in 1) whether they normalize towards decomposed characters (NFD, NFKD) or precomposed characters (NFC, NFKC) and 2) whether the normalization process erases compatibility distinctions (NFKD, NFKC) or not (NFD, NFC).

For use on the Web, it is important not to lose the so-called compatibility distinctions, which may be important (see [UXML] Chapter 4 for a discussion). The NFKD and NFKC normalization forms are therefore excluded. Among the remaining two forms, NFC has the advantage that almost all legacy data (if transcoded trivially, one-to-one, to a Unicode encoding) as well as data created by current software is already in this form; NFC also has a slight compactness advantage and a better match to user expectations with respect to the character vs. grapheme issue. This document therefore chooses NFC as the base for Web-related early normalization.

For a list of programming resources related to normalization, see C Resources for Normalization.

3.2.1 Normalizing Transcoder

A normalizing transcoder is a transcoder that converts from a legacy encoding to a Unicode encoding form and ensures that the result is in Unicode Normalization Form C (see 3.2.2 Unicode-normalized text). For most legacy encodings, it is possible to construct a normalizing transcoder (by using any transcoder followed by a normalizer); it is not possible to do so if the encoding's repertoire contains characters not represented in Unicode.

3.2.2 Unicode-normalized text

Text is, for the purposes of this specification, Unicode-normalized if it is in a Unicode encoding form and is in Unicode Normalization Form C, according to a version of Unicode Standard Annex #15: Unicode Normalization Forms [UTR #15] at least as recent as the oldest version of the Unicode Standard that contains all the characters actually present in the text, but no earlier than version 3.2 [Unicode 3.2].

3.2.3 Include-normalized text

Markup languages, style languages and programming languages often offer facilities for including a piece of text inside another. An include is an instance of a syntactic device specified in a language to include text at the position of the include, replacing the include itself. Examples of includes are entity references in XML, @import rules in CSS and the #include preprocessor statement in C/C++. Character escapes are a special case of includes where the included entity is predetermined by the language.

Text is include-normalized if:

1. the text is Unicode-normalized and does not contain any character escapes or includes whose expansion would cause the text to become no longer Unicode-normalized; or

2. the text is in a legacy encoding and, if it were transcoded to a Unicode encoding form by a normalizing transcoder, the resulting text would satisfy clause 1 above.

3.2.4 Fully-normalized text

Formal languages define constructs, which are identifiable pieces, occurring in instances of the language, such as comments, identifiers, element tags, processing instructions, runs of character data, etc. During the normal processing of include-normalized text, these various constructs may be moved, removed (e.g. removing comments) or merged (e.g. merging all the character data within an element as done by the string() function of XPath), creating opportunities for text to become denormalized. The software performing those operations, or other software down the line that needs to perform normalization-sensitive operations, then has to re-normalize the result, which is a burden. One way to avoid such denormalization is to make sure that the various important constructs never begin with a character such that appending that character to a normalized string can cause the string to become denormalized. A composing character is a character that is one or both of the following:

1. the second character in the canonical decomposition mapping of some character that is not listed in the Composition Exclusion Table defined in [UTR #15], or

2. of non-zero canonical combining class as defined in [Unicode] .

Please consult Appendix B Composing Characters for a discussion of composing characters, which are not exactly the same as Unicode combining characters.

Text is fully-normalized if:

1. the text is in a Unicode encoding form, is include-normalized and none of the constructs comprising the text begin with a composing character or a character escape representing a composing character; or

2. the text is in a legacy encoding and, if it were transcoded to a Unicode encoding form by a normalizing transcoder, the resulting text would satisfy clause 1 above.

Identification of the constructs that should be prohibited from beginning with a composing character (the relevant constructs) is language-dependent. As specified in 3.4 Responsibility for Normalization, it is the responsibility of the specification for a language to specify exactly what constitutes a relevant construct. This may be done by specifying important boundaries, taking into account which operations would benefit the most from being protected against denormalization. The relevant constructs are then defined as the spans of text between the boundaries. At a minimum, for those languages which have these notions, the important boundaries are entity (include) boundaries as well as the boundaries between most markup and character data. Many languages will benefit from defining more boundaries and therefore finer-grained full-normalization constructs.

3.2.5 Normalization-sensitive operations

An operation is normalization-sensitive if its output(s) are different depending on the state of normalization of the input(s); if the output(s) are textual, they are deemed different only if they would remain different were they to be normalized. These operations are any that involve comparison of characters or character counting, as well as some other operations such as ‘delete first character’ or ‘delete last character’.

3.2.6 Text-processing component

A text-processing component is a component that recognizes data as text. This specification does not specify the boundaries of a text-processing component, which may be as small as one line of code or as large as a complete application. A text-processing component may receive text, produce text, or both.

3.2.7 Certified and suspect text

In the following definitions, the word 'normalized' may stand for either 'include-normalized' or 'fully-normalized', depending on which is most appropriate for the specification or implementation under consideration.

Certified text is text which satisfies at least one of the following conditions:

1. it has been confirmed through inspection that the text is in normalized form

2. the source of the text (a text-processing component ) is known to produce only normalized text.

Suspect text is text which is not certified.

3.3.1 General examples

The string suçon (U+0073 U+0075 U+00E7 U+006F U+006E) encoded in a Unicode encoding form, is Unicode-normalized, include-normalized and fully-normalized. The same string encoded in a legacy encoding for which there exists a normalizing transcoder would be both include-normalized and fully-normalized but not Unicode-normalized (since not in a Unicode encoding form).

In an XML or HTML context, the string suçon is also include-normalized, fully-normalized and, if encoded in a Unicode encoding form, Unicode-normalized. Expanding ç yields suçon as above, which contains no replaceable combining sequence.

The string suc¸on (U+0073 U+0075 U+0063 U+0327 U+006F U+006E), where U+0327 is the COMBINING CEDILLA, encoded in a Unicode encoding form, is not Unicode-normalized (since the combining sequence 'c¸' (U+0063 U+0327) should appear instead as the precomposed 'ç' (U+00E7)). As a consequence this string is neither include-normalized (since in a Unicode encoding form but not Unicode-normalized) nor fully-normalized (since not include-normalized). Note however that the string sub¸on (U+0073 U+0075 U+0062 U+0327 U+006F U+006E) in a Unicode encoding form is Unicode-normalized since there is no precomposed form of 'b' plus cedilla. It is also include-normalized and fully-normalized.

In plain text the string suçon is Unicode-normalized, since plain text doesn't recognize that ̧ represents a character in XML or HTML and considers it just a sequence of non-replaceable characters.

In an XML or HTML context, however, expanding ̧ yields the string suc¸on (U+0073 U+0075 U+0063 U+0327 U+006F U+006E) which is not Unicode-normalized ('c¸' is replaceable by 'ç'). As a consequence the string is neither include-normalized nor fully-normalized. As another example, if the entity reference &word-end; refers to an entity containing ¸on (U+0327 U+006F U+006E), then the string suc&word-end; is not include-normalized for the same reasons.

In an XML or HTML context, expanding ̧ in the string sub̧on yields the string sub¸on which is Unicode-normalized since there is no precomposed character for 'b cedilla' in NFC. This string is therefore also include-normalized. Similarly, the string sub&word-end; (with &word-end; as above) is include-normalized, for the same reasons.

In an XML or HTML context, the strings ¸on (U+0327 U+006F U+006E) and ̧on are not fully-normalized, as they begin with a composing character (after expansion of the character escape for the second). However, both are Unicode-normalized (if expressed in a Unicode encoding form) and include-normalized.

The following table consolidates the above examples. Normalized forms are indicated using 'Y', a hyphen means 'not normalized'.

3.3.2 Examples of XML in a Unicode encoding form

Here is another summary table, with more examples but limited to XML in a Unicode encoding form. The following list describes what the entities contain and special character usage. Normalized forms are indicated using 'Y'. There is no precomposed 'b with cedilla' in NFC.

• "ç" LATIN SMALL LETTER C WITH CEDILLA

• "&cedilla;" CEDILLA (combining)

• "&c;" LATIN SMALL LETTER C

• "&b;" LATIN SMALL LETTER B

• "¸" CEDILLA (combining)

• "/" (immediately before 'on' in last example) COMBINING LONG SOLIDUS OVERLAY

3.3.3 Examples of restrictions on the use of combining characters

Include-normalization and full-normalization create restrictions on the use of combining characters. The following examples discuss various such potential restrictions and how they can be addressed.

Full-normalization prevents the markup of an isolated combining mark, for example for styling it differently from its base character (Benoi<span style='color: blue'>^</span>t, where '^' represents a combining circumflex). However, the equivalent effect can be achieved by assigning a class to the accents in an SVG font or using equivalent technology. View an example using SVG (SVG-enabled browsers only).

Full-normalization prevents the use of entities for expressing composing characters. This limitation can be circumvented by using character escapes or by using entities representing complete combining character sequences. With appropriate entity definitions, instead of A´, write Á (or better, use 'Á' directly).