Canonicalization, Normalization

In computer science, canonicalization (sometimes standardization or normalization) is a process for converting data that has more than one possible representation into a "standard", "normal", or canonical form. 

This can be done to compare different representations for equivalence.

URL

A canonical URL is a URL for defining the single source of truth for duplicate content.

XML

XML canonical form, briefly defined, removes whitespace within tags, uses particular character encodings, sorts namespace references and eliminates redundant ones, removes XML and DOCTYPE declarations, and transforms relative URIs into absolute URIs.

source: https://en.wikipedia.org/wiki/Canonicalization

Phone Number

A canonical phone address is a text string with the following structure:

+ CountryCode Space [(AreaCode) Space] SubscriberNumber

For example, +1 (425) 882-8080

source: https://tapiex.com/TPNet_Help/Canonical%20Addresses.htm

Unicode Normalization Forms

Canonical and Compatibility Equivalence

Canonical equivalence is a fundamental equivalency between characters or sequences of characters which represent the same abstract character, and which when correctly displayed should always have the same visual appearance and behavior.

Normalization Forms

The Unicode Normalization Algorithm puts all combining marks in a specified order, and uses rules for decomposition and composition to transform each string into one of the Unicode Normalization Forms.

The four Unicode Normalization Forms are: 

• Normalization Form D (NFD) = Canonical Decomposition
• Normalization Form C (NFC) = Canonical Decomposition, followed by Canonical Composition
• Normalization Form KD (NFKD) = Compatibility Decomposition
• Normalization Form KC (NFKC) = Compatibility Decomposition, followed by Canonical Composition

source: https://unicode.org/reports/tr15/

See also data normalization (TWOTW)

Eszett (ß = German sharp S)

Eszett (ß = German sharp S)

In the German alphabet, ß (Unicode U+00DF) is a consonant letter that evolved as a ligature of "long s and z" (ſz) and "long s over round s" (ſs).

The combination of long s and s is also seen in Early Modern English (example from the US Bill of Rights)

Since the German spelling reform of 1996, both ß and ss are used to represent /s/ between two vowels. In alphabetizing German words, the collation rules say to treat ß as a double "s". 

Source: http://en.wikipedia.org/wiki/%C3%9F

If locale-based string comparison is not available, converting to UPPERCASE is better than lowercase. For example,

uppercase("Straße") == uppercase("Strasse") == "STRASSE"

lowercase("Straße") != lowercase("Strasse"), because "straße" != "strasse"

UPDATE: According to Wikipedia, "Traditionally, ⟨ß⟩ did not have a capital form, although some type designers introduced de facto capitalized variants. In 2017, the Council for German Orthography officially adopted a capital, ⟨ẞ⟩, as an acceptable variant in German orthography, ending a long orthographic debate. Since 2024 the capital ⟨ẞ⟩ (ligature) has been preferred over ⟨SS⟩ (two letters)."

Invariant Culture

The InvariantCulture property can be used to persist data in a culture-independent format. This provides a known format that does not change and that can be used to serialize and deserialize data across cultures.

 

If you are making a security decision (such as whether to allow access to a system resource) based on the result of a string comparison or a case change, you should not use the invariant culture. Instead, you should perform a case-sensitive or case-insensitive ordinal comparison by calling a method that includes a StringComparison parameter and supplying either StringComparison.Ordinal or StringComparison.OrdinalIgnoreCase as an argument.

 

The Turkish ‘i’ Problem

A method call such as IsFileURI("file:") returns true if the current culture is U.S. English, but false if the current culture is Turkish. Thus, on Turkish systems, someone could circumvent security measures that block access to case-insensitive URIs that begin with "FILE:". The Turkish alphabet includes an "I with a dot" character "İ", which is the capital version of "i". Turkish also includes a lowercase "i without a dot" character, "ı", which capitalizes to "I".

 

Source: http://msdn.microsoft.com/en-us/library/system.globalization.cultureinfo.invariantculture%28v=vs.110%29.aspx, http://msdn.microsoft.com/en-us/library/dd465121%28v=vs.110%29.aspx 

Ignorable Characters

A Default Ignorable Character is not rendered by default, but may have an effect on the text. An example of an ignorable character is the soft hyphen (). The appearance of " therapist" is simply "therapist"; no visible glyph. However, the soft hyphen may be displayed as a hyphen at the end of a line.

 

Source: http://www.unicode.org/L2/L2002/02368-default-ignorable.html

Octothorpe

A couple of Bell Labs employees decided to call it an "Octotherp"—a  name pretty much pulled out of thin air. ("Octo-" refers to the shapes  eight lines that stick out of the sides; "-therp" is completely made up.)

"Octotherp" morphed into "octothorpe"—which, rumor has it, came from someone at Bell Labs changing the name to turn it into a tribute to an olympic athlete named Jim Thorpe.  But no one really knows.

source: https://gizmodo.com/before-the-hashtag-there-was-the-octothorpe-1672570850

The symbol # is most commonly known as the number sign, hash, or pound sign. It is believed that the symbol traces its origins to the symbol ℔, an abbreviation of the Roman term libra pondo, which translates as "pound weight". A 1917 manual distinguishes between two uses of the sign: "number (written before a figure)"; and "pounds (written after a figure)".

In the United Kingdom and Ireland, it is generally called a hash (probably ultimately from "hatch", referring to cross-hatching, although the exact derivation is disputed).

It should not be called a sharp, even though it is used in C#. Microsoft says, "It's not the 'hash' (or pound) symbol as most people believe. It's actually supposed to be the musical sharp symbol U+266F (♯). However, because the sharp symbol is not present on the standard keyboard, it's easier to type the hash ('#') symbol.

source: https://en.wikipedia.org/wiki/Number_sign

Listen to Merriam-Webster's Word of the Day "Octothorpe" podcast (2:46)

Additional article on the "#" touch-tone phone key by one of the people who worked on the design for the touch-tone keypad: http://dougkerr.net/Pumpkin/articles/Octatherp-octotherp.pdf