ASCII: What It Is and Where It Is Still Used

You open a text editor and see unfamiliar symbols at the start of a file. Or a colleague sends a message where the text has turned into a string of characters like #5$#@!$&%. Or you look at a hex dump and spot 0D 0A at the end of every line. All of it is a trace of one standard that predates the internet, personal computers and even colour monitors. The standard is called ASCII.

What ASCII is and where it came from

ASCII stands for American Standard Code for Information Interchange. The first version was published in 1963 as ASA X3.4-1963, and the standard was reissued later as ANSI X3.4-1968, -1977 and -1986.

The story starts earlier than that. In May 1961 IBM engineer Bob Bemer proposed to the American Standards Association that a single code be created for exchanging information between machines. The task was practical: every manufacturer had its own character table, and moving data between two machines meant re-encoding it by hand.

The context matters here. The standard was created in the world of teletypes and punched cards, not in the world of text editors. That is where the main technical decision comes from: the code is seven bits wide. Seven bits cover 128 values, and the eighth bit on a paper tape could be given over to a parity check, in other words to detecting errors during transmission. Sending that extra bit down the line was expensive, so it was dropped.

Note the wording: the eighth bit was never part of the ASCII character. It was a separate error-detection mechanism used on telegraph lines. On eight-bit computers without parity, that bit was usually just set to zero. The confusion between «ASCII is seven bits» and «ASCII is eight bits» persists to this day, even though the standard itself is unambiguous.

128 characters and how the table is laid out

Seven bits give exactly 128 code positions, from 0 to 127. The table divides into four groups of 32, and that division is not arbitrary: each group holds exactly 16 printable and 16 non-printable characters, which makes the code easier to navigate than a flat table of 256 positions.

• 0x00–0x1F, codes 0 through 31: control characters. There are 32 of them, and they carry the whole invisible syntax of text formats;
• 0x20–0x3F, codes 32 through 63: space, punctuation and the digits 0 through 9;
• 0x40–0x5F, codes 64 through 95: the @ sign, uppercase Latin letters A through Z and part of the punctuation;
• 0x60–0x7F, codes 96 through 127: lowercase Latin letters a through z, the remaining punctuation and the service code 127.

The table holds 95 printable characters, and those are the numbers from 32 through 126 inclusive. Space has code 32, the exclamation mark 33, the digit 0 has code 48, uppercase A has code 65, lowercase a has code 97, and the last printable character, the tilde, has code 126. Code 127 is not printable: it is DEL, delete, a leftover from the teletype era.

Character  Decimal   Hexadecimal   Binary
space         32           0x20       00100000
!             33           0x21       00100001
0             48           0x30       00110000
A             65           0x41       01000001
Z             90           0x5A       01011010
a             97           0x61       01100001
z            122           0x7A       01111010
~            126           0x7E       01111110

Any number can be converted to hexadecimal or binary with the number base converter, and the binary translator shows how a string looks byte by byte.

Control characters: CR, LF, TAB, ESC

Half the table is taken up by control characters, and they explain why text files contain invisible structures. Each one defines an action rather than a letter.

• NUL, code 0: the null byte, the end-of-string marker in C and the basis of null-terminated strings;
• TAB, code 9: horizontal tabulation, a jump to the next multiple of 8;
• LF, code 10: line feed, the character that programming languages spell as \n;
• VT, code 11: vertical tabulation;
• FF, code 12: form feed or screen clear;
• CR, code 13: carriage return, spelled \r;
• ESC, code 27: escape, the start of terminal control sequences;
• DEL, code 127: delete.

Why is a line feed written as \n when its code is 10? That is no coincidence and no confusion. The sequence \n in C, Java, JavaScript and Python historically corresponds to LF, that is to decimal 10 in the ASCII table. The escape sequence literally means «the character with number 10», and the fact that the decimal spelling looks like an n comes from an older numbering system in use before binary notation became standard, where n sat next to ten.

The second control character that shows up most often is CR with code 13, written as \r. Its purpose is literal: on a teletype, CR returned the carriage to the left margin, while LF moved the paper up one line. Those are two separate mechanisms, and on systems where the printer was the only output device they were issued independently.

Two conventions survive from that. On Unix systems the end of a line is a single LF byte, 0x0A. On Windows the pair CR LF is used, 0x0D 0x0A, where CR returns the carriage and LF drops it to the next line. That is why a file created on one system and read on another sometimes looks like one very long line: the LF goes by unnoticed because the cursor never comes back to the margin.

Unix (LF):        68 65 6C 6C 6F 0A          hello.
Windows (CR LF):  68 65 6C 6C 6F 0D 0A      hello.
Old Mac (CR):     68 65 6C 6C 6F 0D          hello.

0A = line feed, 0D = carriage return
In the text you see: "hello." on one line, "hello" followed by a break

One more control code turns up more often than you would expect, and that is TAB. In aligned text formats such as TSV or the Makefile format, the character with code 9 serves as a column separator or an indent. That is why a file where tabs were replaced with spaces can drift out of alignment when opened in a different program.

ASCII inside modern encodings

ASCII is now used almost exclusively as a foundation. Its first great contribution is that the first 128 code positions of the Unicode table match ASCII symbol for symbol. That is why ASCII counts as a subset of Unicode, and it is why UTF-8 turned out to be backward compatible with ASCII.

The compatibility works like this. A character with a code below 128 is written in UTF-8 as exactly one byte carrying the same value it had in ASCII. A character with code 128 or above takes two, three or four bytes. The upshot is that a file containing nothing above U+007F is byte for byte identical in ASCII and in UTF-8, and a program that only understands ASCII keeps working with new files unchanged. The mechanics of turning code points into bytes are covered in the article about UTF-8 and Unicode.

• ASCII is a subset of Unicode: the first 128 positions match one for one;
• UTF-8 is backward compatible with ASCII: characters below U+0080 take exactly one byte;
• Windows-1251 and KOI8-R also keep ASCII in the first 128 positions and add Cyrillic in the upper half of the table;
• every modern encoding that claims ASCII compatibility keeps letters, digits and basic punctuation in the same places.

Where ASCII shows up today

A standard that is sixty years old has neither vanished nor gone stale. It lives in places where predictability matters more than beauty of representation.

• protocols and headers. HTTP headers, most SMTP commands and the DNS format are built on ASCII, which lets them be parsed byte by byte without elaborate parsers;
• source code and configuration files. Scripts, Makefiles, requirements.txt, nginx and systemd configs are written in ASCII so they behave identically in every environment;
• file names. Restricting names to ASCII avoids whole classes of encoding trouble when files move between systems;
• data formats. CSV, JSON Schema, robots.txt and sitemap.xml all work with a limited character set;
• ASCII art. Banners made of characters, drawings in code comments and in README files, styled terminal output;
• hardware protocols. Many microcontrollers and industrial devices work with restricted character sets over UART;
• teaching material. Morse code, for instance, uses only Latin letters, digits and punctuation, which makes it fit the ASCII table perfectly.

A character banner for terminal output or a README can be produced with the ASCII art generator. Right next to it sits the Morse code translator, which works by the same logic of substituting characters, only with a different set of symbols.

Cyrillic is not in ASCII, and here is what follows from that

The main practical limitation: the table of 128 characters contains not a single Russian letter. Latin letters, digits, punctuation and control codes, that is the whole inventory. Historically this was fine, since the standard was created in an English-speaking environment, but today it produces concrete problems.

The first option, adding Cyrillic to the table itself, was rejected: a standard is a standard, and it does not get rewritten on the fly. The second option, extending the table in an eight-bit version, worked, and that is where encodings such as Windows-1251 and KOI8-R came from, each keeping ASCII in the first 128 positions and putting Cyrillic in the upper half.

Code   Character   ASCII at this position
0xC0      А         not defined
0xD8      Ё         not defined
0xDF      Я         not defined
0xE0      а         not defined
0xF8      ё         not defined
0xFF      я         not defined

Positions 0x00–0x7F in Windows-1251 match ASCII exactly.

The third option, a table of 1 114 112 positions with characters stored in UTF-8, is the solution everyone arrived at. It brought a side effect along, though: if a program reads a file as ASCII, it does not merely fail to see the Cyrillic, it can spoil the entire file. One Russian letter in UTF-8 occupies two bytes, and if one of those bytes lands on a control code, the result is genuinely strange.

Unreadable characters in a file: where they come from

A topic worth its own section is files where strange symbols appear where text should be. There are several causes and they are worth telling apart, because each is fixed differently.

• a file in UTF-16 read as single-byte. Nul bytes show up between characters, because every character in UTF-16 takes two bytes;
• a UTF-8 file with a BOM, where the first three bytes EF BB BF get interpreted as separate characters and leave an invisible tail at the start;
• a UTF-8 file without a BOM read as windows-1251. You get the classic picture of two Latin letters standing in for one Russian one;
• a binary file opened in a text editor. There genuinely is no text in there, only bytes;
• a file where line endings turned into non-breaking spaces or a non-breaking hyphen after copying from the web.

The encoding detector identifies the encoding from the leading bytes and lets you compare the reading variants side by side.

Reading a dump without confusing bytes with characters

A hex dump shows bytes, not characters, and that is the main source of confusion. Each byte occupies two hexadecimal digits in a dump, because a byte holds exactly 256 values while a hex digit gives 16. One byte always fits into exactly a pair of hex digits, which is what makes dumps readable at a glance.

To take such values apart, the HEX to RGB converter helps: it shows the decimal value of every byte and its share of the colour, and along the way demonstrates the rule that one byte equals two hex digits.

One more thing about reading dumps: how escape sequences appear. When program source contains \t, that is not the two characters backslash and t, it is one character with code 9. The same holds for \n, \r and every other named code. The difference is visible at the byte level: in a file an escape sequence occupies two bytes, while the control character itself occupies one.

ASCII against Unicode: where the difference lies

The two terms get mixed up constantly even though they answer different questions. ASCII is a specific table of 128 characters with a specific set of control codes. Unicode is an agreement about which characters exist in the world at all, with numbers running from U+0000 to U+10FFFF. UTF-8 is the way of packing those numbers into bytes.

• ASCII: 128 characters, 7 bits, Latin and control codes only, a standard from 1963;
• Unicode: more than 1.1 million code positions, every script in the world, a standard since 1991;
• UTF-8: an encoding where a character takes one to four bytes depending on its number;
• ASCII sits inside Unicode as a subset, so any program that handled ASCII keeps working with UTF-8 for Latin text.

The practical conclusion: when the question is what those characters in a particular file actually are, start by identifying the encoding rather than guessing from appearance. ASCII remains a useful reference point to count from.

Typical mistakes

• believing ASCII is eight bits wide. The standard is seven bits, and the eighth was used for parity checking on paper tape;
• confusing 95 with 128. There are 95 printable characters, codes 32 through 126, and 33 control characters, codes 0–31 and 127;
• forgetting about case in the letter codes. Uppercase A is 65, lowercase a is 97, a difference of exactly 32;
• assuming \n is the letter n. It is a control character with code 10, and in a dump it occupies one byte whereas the escape sequence in source occupies two;
• expecting an ASCII file with Cyrillic to work. There is no Cyrillic in the standard at all;
• writing a character code into HTML without checking that it fits in ASCII. Codes above 127 have to be written as numeric references or named entities, otherwise the markup breaks.

Common questions

How is ASCII different from UTF-8?

In size and in reach. ASCII describes 128 characters in seven bits, 95 of them printable, and nothing but Latin. UTF-8 describes over a million Unicode code points using one to four bytes per character. ASCII is fully contained in UTF-8, and text containing nothing above U+007F looks identical in both encodings.

Why does ASCII only have 128 characters?

History and technology. The standard was built for telegraph lines and punched cards, where every character cost a separate bit of transmission time, and economy mattered more than convenience. The eighth bit was given over to parity checking during transmission, so it was never part of the character itself. Adding characters to the table would have meant changing the standard, which turned out to be easier to achieve through separate extensions.

What does 0x20 mean and why is it not 32?

0x20 is simply the same number written in hexadecimal: 2 times 16 plus 0, which comes to 32 in decimal. Different notations for one value. File dumps conventionally show bytes in hex because they line up in columns and read in pairs, while decimal values are easier to work out in your head.

Why does \n mean 10 and not the letter n?

Because the sequence \n stands for the character with decimal number 10, which is line feed in the ASCII table. The letter n sits next to ten in an older numbering system that was in use before binary notation spread, and that coincidence became fixed in the syntax of programming languages. It comes with company: \t is code 9, \r is code 13, \0 is code 0.

Can Cyrillic be written in ASCII?

No, the table holds not a single Russian letter. Trying to put one there produces either an error or a completely different character. Russian text needs an encoding that has Cyrillic: Windows-1251, KOI8-R, or, which is the better choice today, UTF-8. The difference between Unicode and UTF-8 and the ways to avoid garbled text are covered in a separate article.

Why does my file show odd characters instead of text?

Most likely the file is being read in a different encoding than the one it was written in. The encoding detector identifies the encoding from the first bytes and lets you compare readings. If what you see are genuinely unprintable characters, the file is probably binary and a text editor is the wrong tool for it.