UTF-8 and Unicode: What Sets Them Apart and Why Text Turns to Garbage

You open a CSV export and the surname column reads something like «РџСЂРёРІРµС‚». A colleague sends a document where the Russian words have turned into «Ð¡Ñ‚рика». Or the other way round: opening the file in an editor gives you a row of question marks instead of letters, and there is no telling what was actually inside. All of it is the same disease. The file was written in one encoding and read in another. It turns out to be far easier to untangle than it looks, provided you start from one question: what is the difference between ASCII and UTF-8.

Unicode is the table, UTF-8 is the writing system

The two words get used as synonyms all the time, and that is where most of the confusion starts. They answer different questions. Unicode answers «which characters exist at all», while UTF-8 answers «how do those characters get written into bytes». Think of a street address and a postal code: the address stays the same, only the delivery method changes.

• Unicode is the standard that catalogues the characters of human scripts, punctuation, emoji and mathematical notation. Each one gets a number, called a code point, written like U+0410;
• UTF-8 is an encoding, that is, the rule that turns a code point into a sequence of bytes. The same code point can be written several ways: UTF-8, UTF-16, UTF-32;
• ASCII is the older 1963 standard with exactly 128 characters: Latin letters, digits, punctuation and control codes.

The phrase «re-encode the file to Unicode» is as meaningless as «rewrite the address as a postal code». You have to say which one: UTF-8, UTF-16 or UTF-32. But since ASCII is also a table of characters rather than an encoding, how does it relate to UTF-8? As a subset: the first 128 positions of the Unicode table match ASCII symbol for symbol.

U+0041   code point 65,     the character «A»
U+0410   code point 1040,   the character «А» (Cyrillic)
0x41     the same number 65 in hexadecimal
41       the same 65 in decimal, no prefix

How many bytes a character takes

In ASCII every character took exactly one byte, because there were only 128 possible values and a byte holds 256. UTF-8 cannot work that way: Unicode covers well over two hundred thousand characters, while the byte is still one byte long. The length of a character in UTF-8 therefore depends on its code point, and you cannot guess it in advance.

• one byte: code points U+0000 to U+007F, 128 values: all of Latin, digits, basic punctuation and space;
• two bytes: U+0080 to U+07FF, 1920 values: Cyrillic, Greek, Hebrew, accented Latin;
• three bytes: U+0800 to U+FFFF, 63488 values: ideographs, arrows, rare letters;
• four bytes: U+10000 to U+10FFFF, 1048576 values: emoji, ancient scripts, music symbols.

Cyrillic lands in the second row, which is why a Russian letter costs two bytes in UTF-8 and not one. A practical consequence surprises people: Russian text in UTF-8 weighs roughly twice as much as the same text in Windows-1251, where every letter, Latin or Cyrillic, costs exactly one byte.

A     U+0041   41           1 byte
А     U+0410   D0 90        2 bytes
€     U+20AC   E2 82 AC     3 bytes
𝄞    U+1D11E  F0 9D 84 9E  4 bytes

Привет → D0 9F D1 80 D0 B8 D0 B2 D0 B5 D1 82   (12 bytes, 6 characters)

Notice the rhythm: the bytes D0 and D1 come back almost every second position, with D0 9F being П, D1 80 being р and D0 B8 being и. Because the pattern is so regular, Russian text in UTF-8 is recognisable in a hex dump at a glance, before you have even worked out what you are looking at.

Bytes inside files are often written in hexadecimal because that is easier to read. The pair 0xD0 0x9F then becomes «D0 9F», and three bytes in a row become a colour such as #D09FD1. The HEX to RGB converter takes a hexadecimal value apart into its components, and along the way it shows why one byte always fits into exactly two hex digits.

ASCII lives inside UTF-8

Understanding how ASCII relates to UTF-8 matters, because «what is the difference between ASCII and UTF-8» is one of the most frequent questions around. The answer is short: ASCII is a subset of UTF-8. All 128 ASCII characters sit between 0 and 127, that is, in the first row of the table above, and in UTF-8 they occupy exactly one byte each with the same values. The table itself, its control codes and the history of the standard are covered separately in the article about ASCII and its 128 characters.

• byte 0x41 in ASCII and byte 0x41 in UTF-8 are the very same letter A;
• any file containing nothing above U+007F is byte-for-byte identical in both encodings;
• so pure ASCII text moved into UTF-8 is indistinguishable from the original and survives the move intact;
• UTF-8 was designed so that old programs able to read only ASCII would keep working with new files, which was the whole point of the design.

That also answers a common question: can Russian text be written in ASCII? No. ASCII contains not a single Cyrillic letter, and trying to put one there produces either an error or a completely different character. The smallest set covering Russian, Latin and digits is UTF-8 or Windows-1251, and the second one now survives only as legacy.

The BOM and why it breaks things

The BOM, byte order mark, is a marker at the very beginning of a file telling the reader «the text that follows is in such and such encoding». For UTF-8 it is three bytes: EF BB BF. On its own the marker is useful when there is no other way to determine the encoding, but UTF-8 has a specific problem with it. Byte order inside UTF-8 is always the same, so the three extra bytes at the start carry no information while still breaking plenty of things.

• EF BB BF at the start of a file means it was written as UTF-8 with a BOM;
• JSON is not supposed to begin with a BOM, and strict parsers reject such a file with an unexpected character error;
• the BOM ends up inside the result when a program reads the file as plain text without stripping the marker, so the first field starts with an invisible character and stops matching its key;
• a Content-Type header of text/plain; charset=utf-8 must not carry a BOM either, though in a file served as an attachment the marker sometimes survives and shows up as an invisible character at the start of the document.

EF BB BF     UTF-8 with BOM
FF FE        UTF-16 LE
FE FF        UTF-16 BE
00 00 FE FF  UTF-32 BE

The rule is simple: for JSON, CSV, HTML, config files and source code, use UTF-8 without a BOM. Editors such as VS Code write it that way by default, which is the right choice. Keep a BOM only where the file is passed on as-is and the marker is what tells the receiving side how to read it.

Mojibake: where garbage text comes from

The word «кракозябры» was coined for the moment when Russian text inside a file turns into something like «РџСЂРёРІРµС‚». Technically this is called mojibake, and the mechanism is dead simple: the bytes are exactly the same, they are just being looked up in a different table.

The word «Привет» takes 12 bytes in UTF-8. Read those 12 bytes as Windows-1251 and you get 12 characters, because in a single-byte encoding every byte stands on its own. That is how garbled text is born.

UTF-8, 12 bytes:        D0 9F D1 80 D0 B8 D0 B2 D0 B5 D1 82
read as CP1251:         Привет
read as KOI8-R:         п═я÷п║п┌п═я▒п═п├п═б╣п║Б─

• the file is UTF-8 but the program reads it as Windows-1251, the most common case, and the source of the classic picture with the letter Р;
• the file is UTF-16LE while the program expects a single-byte encoding, which produces characters like «@8» mixed with unprintable noise;
• double encoding: the text was already converted to UTF-8, then read as CP1251 and written back out as UTF-8, so it is spoiled twice over;
• the file is UTF-8 but the HTML never declares an encoding, so the browser guesses KOI8-R or windows-1252;
• the text went through a clipboard or a messenger and came out as question marks, and in that case some of the information is gone for good.

The different cases look different. When one Russian letter turns into two Latin or exotic ones, the file is single-byte and is being read as UTF-8. When you get empty squares, question marks or diamonds instead of letters, the characters are simply not representable in the encoding the program is using, and the information is already lost.

How to determine the encoding of a file

1. open the file in a hex editor and look at the first bytes: EF BB BF means UTF-8 with a BOM, FF FE or FE FF means UTF-16, and nothing like that means a single-byte encoding;
2. check whether any byte above 127 appears: if not, the file is pure ASCII and the encoding hardly matters;
3. if such bytes exist, try reading the file as UTF-8: if the text comes out cleanif the text comes out clean, with no garbage and no question marks, you have found the encoding;
4. if UTF-8 produces nonsense, try Windows-1251 and KOI8-R and see which one yields readable Russian;
5. sanity-check the result against what the file is: an export from 1C or Excel is most likely windows-1251, while a downloaded or editor-created file is usually utf-8.

All of this takes a couple of minutes by hand, but for a quick check there is an encoding detector: it takes text or a file, works out the encoding from the leading bytes and the byte statistics, flags a BOM separately, and shows the same text as UTF-8 and as CP1251 side by side. To look at bytes and get them back, a binary translator helps, and to see text as a safe string for a URL or a JSON payload, there is a Base64 tool.

Transcoding without losing anything

Once you know the encoding, a file can be converted. It takes a single command, and it matters which way round you run it: the flag -f sets the encoding you read from, the flag -t sets the one you write.

# windows-1251 → utf-8, the original stays untouched
iconv -f CP1251 -t UTF-8 input.csv > output.csv

# the same in Python, without losing the last line
python3 -c "import sys; sys.stdout.write(open('input.csv',encoding='cp1251').read())" > output.csv

On Windows PowerShell the built-in equivalents work too: read the bytes of the file, then write them back as text in the encoding you want, telling the writer that no BOM should be added. The plain Notepad in Windows offers the same thing through «File» → «Save as» → the little arrow under the Save button → «Encoding» → UTF-8, with the «Add BOM» tick removed.

The one rule during transcoding: never overwrite the original. If something goes wrong and the source file has already been replaced, there is no way back. Double encoding is exactly the failure mode of a file that was «converted once and came out converted twice».

Why UTF-16 and UTF-32 exist

• UTF-16 uses two bytes per character for everything inside the Basic Multilingual Plane (up to U+FFFF) and four bytes for characters outside it, such as emoji, through surrogate pairs. The byte order within a character can vary, which is why a file without a BOM does not always read back correctly;
• UTF-32 gives every character exactly four bytes whether it is Cyrillic or an emoji. The size is predictable, but you pay for it in volume: Russian text in UTF-32 takes four times as much space as in UTF-8;
• UTF-16 is built into Windows: text files saved as «Unicode», some XML documents and internal formats of office suites;
• the web barely sees UTF-16 at all, since HTTP headers, JSON and HTML all assume UTF-8;
• older systems where memory was expensive often picked UTF-16 as a two-byte compromise, an argument that no longer holds.

In practice: JSON, CSV and the web

• JSON must be UTF-8 according to RFC 8259. Values need no escaping, but a BOM at the start is not allowed, and some parsers reject the whole file because of it;
• CSV has no encoding of its own and no standard for one: the format only settled on delimiters and quoting. That is why windows-1251 still turns up in CSV files so often, and why a BOM is often placed on the first line so that Excel opens the file correctly;
• in HTML the encoding is declared with a meta charset tag set to utf-8 within the first 1024 bytes of the document, otherwise the browser may guess wrong;
• in an HTTP header the encoding travels as the charset parameter of Content-Type: text/html; charset=utf-8;
• the encoding of the database and the encoding of the connection are different things. A MySQL instance storing Latin1 with a utf8mb4 client produces garbled text no matter what the HTTP header says.

Typical mistakes

• assuming Unicode and UTF-8 mean the same thing and hunting for a «Unicode» entry next to «UTF-8» in a settings list;
• opening a file in an editor that silently reassigned the encoding and saving, which replaces some characters with question marks;
• double encoding: a file was read as Windows-1251 and written back as UTF-8 even though it was already UTF-8;
• declaring charset=utf-8 in the HTML while the server or the CMS actually sends windows-1251;
• keeping a BOM inside a JSON file and forgetting that the keys pick up an invisible character too;
• expecting a byte count to equal a character count: Russian UTF-8 is twice as long, and that is normal.

Common questions

Are UTF-8 and Unicode the same thing?

No, they sit on different levels. Unicode is the agreement about which characters exist and what number each one carries. UTF-8 is one of the ways of packing those numbers into bytes, and the most economical one for Latin and Cyrillic. A workable analogy: Unicode is the language, UTF-8 is the alphabet it gets written in.

How many bytes does a Russian letter take?

Two bytes in UTF-8, because Cyrillic sits in the code point range U+0080 to U+07FF. One byte in Windows-1251, two in UTF-16, four in UTF-32. Upper and lower case weigh exactly the same: both «а» and «А» take two bytes.

Should I remove the BOM?

For JSON, CSV, HTML, config files and source code, yes, write without it. The marker tells a UTF-8 reader nothing, but it adds three bytes that confuse parsers, break comparisons of the first line, and hide at the start of the first key. Keep a BOM only where it is the sole signal of the encoding, which usually means a file handed over untouched.

Why does my file open as garbled text?

Because the encoding used for writing and the encoding used for reading did not match. Check three things: whether a BOM sits at the start, whether the file holds bytes above 127, and what the encoding detector reports when it tries the file as UTF-8, windows-1251 and KOI8-R. If the Russian text turned into question marks rather than exotic letters, the original text is already gone and you will need another copy of the file.

How do I check a file encoding in one click?

Drop the file into the encoding detector: it shows the most likely candidate, flags a BOM separately, and puts the same text as UTF-8 and as CP1251 next to each other so you can see the difference with your own eyes. When you already know what the content should say, you can also identify the encoding without any service at all, just from the first three bytes.

What do I do with windows-1251 files?

Work out where they are needed and convert them once to UTF-8 with iconv or any editor that can save in a chosen encoding. Old exports from 1C and Excel are worth converting on the way in, while data stored in a database is better kept in utf8mb4 from the start. And always keep the original: transcoding without a copy is an operation with no undo.