To C#-3 Notes

• An encoding is a binary representation of a character for storage in a disk file.

• An historically important encoding scheme is the ASCII (American System for Computer Information Interchange) system, which was proposed in 1963 by engineers at IBM.

• The 7-bit ASCII (128 characters) character set as it is used today was first published in 1967 and last revised in 1986.

  

• An extension to ASCII was the OEM (Original Equipment Manufacturer) extended ASCII character set (about 1969). It was the character set used on the original IBM PC.

  

• The OEM character set used an 8-bit (256 characters) representation.

• Many other extended ASCII character sets were proposed, many of them containing non-roman language characters. Here are a few from a long list: 1250 (Eastern Europe), 1251 (Cyrillic), 1252 (Western European), 1253 (Greek), 1254 (Turkish).

• In 1987, the term Unicode was introduced. It referred to a comprehensive character set for encoding the alphabets of all major languages (both ancient and modern) in the world. See http://www.unicode.org for details.

• Unicode only specifies character encodings, not specific fonts. In order to display a character, a font set that includes that character may need to be loaded onto that computer.

• Whereas most official Unicode codes only require two bytes, some require only one byte and some even require four bytes.

• The language that requires some 4-byte Unicode codes, called supplementary characters, is Chinese. Here is an article that supplies some data for testing whether your browser can display Unicode supplementary characters:
  www.i18guy.com/unicode/supplementary-test.html

• Several encoding schemes are used to store Unicode codes.
  1. UTF-8   This encoding is very commonly used. It is good for files that contain mostly 1-byte characters but occasionally 2-byte or 4-byte characters. It uses the following scheme:
     

      Min Hex  Hex Max   Binary Byte Sequence
     00000000 0000007F 0vvvvvvv
     00000080 000007FF 110vvvvv 10vvvvvv
     00000600 0000FFFF 1110vvvv 10vvvvvv 10vvvvvv
     00010000 001FFFFF 11110vvv 10vvvvvv 10vvvvvv 10vvvvvv
     00200000 03FFFFFF 111110vv 10vvvvvv 10vvvvvv 10vvvvvv 10vvvvvv
     04000000 7FFFFFFF 1111110v 10vvvvvv 10vvvvvv 10vvvvvv 10vvvvvv 10vvvvvv
     

     For example, the greek character s (sigma) has the hex Unicode code 03C3, which is 0000 0011 1100 0011 in binary. 03C3 is between 0080 and 07FF, so in UTF-8, this becomes 11001111 10000011 = 1100 1111 1000 0011 = CF 83 hex. The red bits are from the unicode code 011 1100 0011 and the green bits are the special UTF-8 designations.

  2. UTF-16   UTF-16 comes in Big Endian and Little Endian versions. The letter s is stored as 03 C3 in Big Endian (high order bytes first) but as C3 03 in Little Endian (low order bytes first). Special provisions are made to encode 4 byte Unicode codes.

  3. UTF-32   UTF-32 also comes in Big Endian and Little Endian versions. The letter s is stored as 00 00 03 0C in Big Endian and as 0C 03 00 00 in Little Endian.

• An alternative is to record the encoding at the beginning of the file itself using Byte Order Marks (BOM). Here is a table of some common BOMs:

  UTF-8	EF BB BF		UTF-32 Big Endian	00 00 FE FF
  UTF-16 Big Endian	FE FF		UTF-32 Little Endian	FF FE 00 00
  UTF-16 Little Endian	FF FE

• In earlier versions, .Net supported more encodings, which the various versions of UTF have mostly replaced. This Console app C# program displays some of the legacy encoding types that .Net still supports.