ASCII: a working reference#

ASCII is the character encoding that almost everything else is built on top of. It defines 128 code points, numbered 0 to 127, and it has not changed since 1986. Learn it once and you have the bottom layer of text handling in every language, every protocol and every file format you will meet on a computer science course.

This page is the whole standard plus the parts that matter in practice: the tables, the bit tricks, the control characters nobody explains, the extended 8-bit code pages that came after, and how all of it relates to Unicode and UTF-8.

Quick answers#

The four numbers worth memorising:

Everything else can be derived. Z is A plus 25, so 90. z is 122. 9 is 57. The letters and digits are each one unbroken run, which is why range checks such as c >= 'a' && c <= 'z' are correct in ASCII (and why they are not correct in EBCDIC, where the alphabet has gaps).

The chart#

The classic layout. Read a code point by combining its column (the high 3 bits) with its row (the low 4 bits), so K sits in column 4, row B, which is 0x4B, decimal 75. Every structural property of ASCII is visible in this grid: the two control columns on the left, the two letter columns with uppercase directly above lowercase, and the digits stacked in column 3.

How ASCII is laid out#

Seven bits, 128 code points#

ASCII is a 7-bit code. That is the single most important fact about it, and it explains the size of the table: 2 to the power 7 is 128, so the valid code points are 0 to 127 and nothing else.

Seven bits was a deliberate compromise. Six bits (64 characters) could not hold both cases of the alphabet plus digits and punctuation. Eight bits would have been wasteful on hardware where a character cost real money to store and transmit, and the eighth bit was wanted anyway: serial links used it as a parity bit for error detection.

The consequence you will hit in code is that a byte can hold values 128 to 255 that are not ASCII at all. "Extended ASCII" is not a standard, it is a family of mutually incompatible guesses about what those extra 128 values mean. See Beyond 7 bits.

The four columns of 32#

Split the table into four blocks of 32 and the design becomes obvious:

Block 2 and block 3 are the same block with one bit changed. Block 0 is block 1 or block 2 with the top bits cleared. None of this is coincidence, and all of it is exploitable.

The case bit#

Bit 5, worth 32, is the only difference between an uppercase letter and its lowercase partner:

A = 0100 0001 = 65
a = 0110 0001 = 97
         ^
     bit 5, value 32

So for letters only:

c | 0x20    /* force lowercase */
c & ~0x20   /* force uppercase, that is c & 0xDF */
c ^ 0x20    /* swap the case */

The same 32 gap explains a subtler thing: because uppercase letters all have smaller code points than lowercase ones, a plain byte-order sort puts Zebra before apple. That is ASCIIbetical order, and it is not alphabetical order. Case-insensitive sorting is a deliberate extra step.

The digit trick#

Digits occupy 48 to 57, and their low four bits are the value of the digit:

'0' = 0011 0000 = 48    low nibble 0000 = 0
'7' = 0011 0111 = 55    low nibble 0111 = 7
'9' = 0011 1001 = 57    low nibble 1001 = 9

Which gives the two conversions you will write a hundred times:

int value  = c - '0';    /* character to number */
char digit = n + '0';    /* number to character */

That is the entire body of a simple atoi, and it is why c & 0x0F also works on a digit. Hexadecimal needs one extra step, because the letters are not adjacent to the digits:

int hexval(char c) {
    if (c >= '0' && c <= '9') return c - '0';
    if (c >= 'a' && c <= 'f') return c - 'a' + 10;
    if (c >= 'A' && c <= 'F') return c - 'A' + 10;
    return -1;
}

The control key trick#

Control characters are not an arbitrary list bolted on to the front of the table. A control character is its printable partner with the top two bits cleared, which in hardware terms is what the Ctrl key on a terminal keyboard physically did:

C   = 0100 0011 = 67
^C  = 0000 0011 =  3   (ETX)

So Ctrl plus a key gives you key & 0x1F, or equivalently key - 64 for uppercase letters. Read the table the other way and every control character has an obvious keystroke:

• Ctrl+I is 9, which is Tab. Tab and Ctrl+I are literally the same byte, which is why they are interchangeable in a terminal.
• Ctrl+M is 13, carriage return, which is what the Enter key sends.
• Ctrl+[ is 27, Escape.
• Ctrl+H is 8, backspace.

This is also why terminal shortcuts cluster where they do. Ctrl+C, Ctrl+D, Ctrl+S and Ctrl+Z all have their behaviour because of what the corresponding control character meant.

The full tables#

Four tables of 32 rows, matching the four blocks above. Every code point in ASCII appears exactly once across them.

Control characters: 0-31 and 127#

Codes 32-63: space, punctuation and digits#

Codes 64-95: uppercase and symbols#

Codes 96-127: lowercase, symbols and DEL#

Control characters explained#

Thirty-three of the 128 code points print nothing. They were commands to a teleprinter or a communications link, and most are dead, but the survivors are load-bearing: you cannot write a network protocol, a terminal program or a file parser without meeting several of them.

Line endings, the one that actually bites#

CR (13) and LF (10) are two separate characters because a teleprinter needed two separate motions: return the carriage to the left margin, and advance the paper by one line. Operating systems then disagreed about which to keep.

Practical consequences:

• A file written on Windows and read on Linux has a trailing \r on every line. It is invisible in most editors and it breaks string comparisons, so if line == "yes" fails on "yes\r". Strip with line.rstrip("\r\n") in Python or TrimRight equivalents elsewhere.
• Git normalises line endings if you ask it to. * text=eol=lf in .gitattributes is the usual fix, and it stops a diff showing every line of a file as changed.
• Opening a file in text mode in C or Python on Windows silently translates CRLF to \n on read and back on write. Opening in binary mode ("rb") does not. Mixing the two is a classic bug.
• HTTP headers are terminated by CRLF, and the header block ends with a bare CRLF CRLF. If you are writing a socket server by hand, sending \n alone will work with lenient clients and fail with strict ones.

The four separators nobody uses#

Codes 28 to 31 are FS, GS, RS and US: a four-level hierarchy of delimiters, built into ASCII from the start, and almost entirely ignored.

US (31)  separates fields   inside a record
RS (30)  separates records  inside a group
GS (29)  separates groups   inside a file
FS (28)  separates files    inside a stream

The point is that these characters never appear in ordinary text, so a format built on them needs no quoting, no escaping and no rules about commas inside values. Every CSV parsing bug in history exists because the world picked a comma, which appears in real data, over US, which does not. They are still used in GS1 barcodes and some EDI and point-of-sale formats, and they are a reasonable choice for a quick internal data dump.

Characters that are whitespace#

Six ASCII code points count as whitespace for isspace in C and for \s in most regular expression flavours:

Note what is not in that list: NUL is not whitespace, and neither is the no-break space at 160, which is not ASCII at all but turns up constantly in text copied from web pages and word processors. It looks exactly like a space and fails every equality test against one.

Printable characters worth knowing#

Converting between representations#

Doing it by hand#

Hex is the natural way to write ASCII because one hex digit is exactly four bits, so a byte is always two digits and the split lines up with the structure of the table. Octal is a leftover from machines with 12, 18 and 36 bit words, and it survives in C escapes and in Unix file permissions.

binary   0100 0001
hex         4    1     = 0x41
decimal  64 + 1        = 65
octal    001 000 001   = 0101   (group in threes from the right)

To go from decimal to binary quickly, subtract the powers of two from the left:

75 - 64 = 11   so bit 6 is set   (0100 0000)
11 - 8  = 3    so bit 3 is set   (0000 1000)
3  - 2  = 1    so bit 1 is set   (0000 0010)
1  - 1  = 0    so bit 0 is set   (0000 0001)
                          total   0100 1011 = 'K'

Doing it in code#

Useful command line one-liners:

# Show the bytes of a file, with printable characters alongside
xxd file.txt | head

# Same idea, named characters, good for spotting stray control bytes
od -c file.txt | head

# Look up a single character
python3 -c "print(ord('A'))"

# Print the whole printable range
python3 -c "print(''.join(chr(i) for i in range(32,127)))"

# The manual page that ships with most Unix systems
man 7 ascii

# Find non-ASCII bytes in a file
grep -nP '[^\x00-\x7F]' file.txt

Beyond 7 bits: the extended code pages#

Once bytes were reliably 8 bits, the parity bit became free real estate, and everyone filled codes 128 to 255 differently. There is no such thing as "extended ASCII" as a single standard. If a file contains a byte above 127, you cannot know what character it means without being told the encoding.

The three sets below are the ones you are most likely to meet. All three agree exactly with ASCII for bytes 0 to 127, which is the only reason mixed-encoding text is ever partially readable.

ISO 8859-1, also called Latin-1#

The ISO standard for Western European languages, and for a long time the default assumption for HTTP and for many databases. Bytes 160 to 255 are accented letters and common symbols. Bytes 128 to 159 are the C1 control range and are not printable characters at all.

Latin-1 has one property that makes it special: its 256 characters map one to one onto the first 256 Unicode code points. Decoding arbitrary bytes as Latin-1 therefore never fails, which makes it a useful last resort for reading a file of unknown encoding without an exception, and a dangerous default because it silently produces nonsense rather than an error.

Windows-1252#

Microsoft's near-identical variant, and the real encoding behind a great deal of text that claims to be Latin-1. It matches ISO 8859-1 everywhere except 128 to 159, where instead of C1 controls it puts typographic characters: curly quotes, the en dash and em dash, the bullet, the ellipsis and the euro sign.

This single difference causes most of the mojibake on the web. A document is labelled ISO-8859-1, actually contains Windows-1252, and its curly quotes decode as invisible control characters or as replacement characters. The HTML5 specification gave up and requires browsers to decode anything labelled ISO-8859-1 as windows-1252 instead.

Code page 437#

The character set burned into the ROM of the original IBM PC. It uses the upper 128 for accented letters, Greek letters, mathematical symbols and, most famously, the box-drawing and shading characters that every DOS-era text interface was built from. Anyone who has seen an old installer's blue screen with double-line borders has seen CP437.

It is worth knowing for two reasons: it is the encoding of a great deal of historical material, including ANSI art and the text-mode games of the period, and its box-drawing characters were all adopted into Unicode, so you can still use them today.