XML: How to Validate It and Turn It into JSON

People end up with XML for all sorts of reasons. An export from an accounting system, a sitemap for a search engine, an RSS feed, a build configuration, a response from a web service: it all arrives as XML, while half of the applications on the other end expect JSON. After that comes the unpleasant part. The file will not open because one closing tag is missing, and the log says nothing about it. Let us look at how both formats are built, how to find an XML error in ten seconds, and how to convert one into the other without losing data on the way.

How XML differs from JSON

XML is a markup language: text where the meaning is carried not by the words themselves but by tag names. JSON is a data format: a structure of keys and values where the type of every value is known in advance. Here is the same record written twice.

XML:
<person><name>Ivan</name><age>34</age></person>

JSON:
{"name":"Ivan","age":34}

The difference is visible at once. In XML the age is the text "34", and unless something states that it is a number, it stays text. In JSON 34 is a number and you can add it to something. In XML there are attributes, which carry properties rather than values, while in JSON properties get their own keys. XML has no built in types for true, false and null: a boolean or an empty value has to be described by hand or fixed in a schema, while in JSON these types exist by definition.

• XML is strict: tag names, letter case, and the quotes around attribute values are all checked.
• JSON is strict about how it is written but forgiving about meaning: only double quotes, no comments, no trailing comma.
• XML has comments, an XML declaration, processing instructions and CDATA sections. JSON has none of those.
• In XML a repeated element simply repeats, while in JSON elements with the same name are collected into an array.
• In XML there are namespaces, so one and the same name can belong to different vocabularies. JSON has no such problem.

When to use XML and when to use JSON

In practice the choice comes down to who picked the format first. If you are the one picking, take JSON: it is more compact, it parses without setup, and it is understood the same way in a browser, in a mobile app and in server code.

• JSON when data travels between your code and a browser or a mobile app.
• JSON when the answer is delivered as an API and parsing speed matters.
• XML when the format is imposed from outside: a sitemap, RSS, an export, a report, a document that carries an electronic signature.
• XML when you need comments, a meaningful element order, or binding to an XSD schema.
• If the data arrives as a table rather than a tree, bring it to JSON first using the guide about converting CSV to JSON, and work with an object from there.

Checking XML for errors: what a validator looks for

An XML error almost never looks like a compiler message. Usually it is a fragment somewhere in the middle of a large file where a closing bracket got lost or where an attribute has an unpaired quote. The check comes down to walking the file through its tags and making sure they close in the right order.

• An unclosed tag: <a><b>text</a>. The closing tag has to match the most recently opened one.
• An extra closing tag: </div> with no matching <div>.
• Missing quotes around an attribute value: <a href=page.html>. In XML the quotes are required, and the character < is not allowed inside the value.
• An unescaped ampersand: <p>Price & delivery</p>. In XML the character & always starts an entity, so it has to be written as &amp;.
• A missing slash on an empty element: <br> instead of <br/>.
• An empty document, or text sitting outside the root element.

<catalog>
  <book id="1">
    <title>Drones</title>
    <price currency="RUB">1990</price>
  </book>
  <book id="2">
    <title>Ropes</title>
  </catalog>

Here is what is wrong: the tag <book id="2"> was opened and never closed, and </catalog> closed it instead. The correct version puts </book> before </catalog>. The formatter catches this and points at the unexpected closing tag. The procedure is simple: paste the file into the XML formatter and press Format. The tag pairing takes a second and there is little room for error.

What this check catches and what it misses

It is worth not overestimating what is available. The tag pairing check finds an unclosed tag and an extra closing tag, but it lets through missing quotes around an attribute value, an unescaped ampersand and a typo in an attribute name, because none of those break the structure. Real validation is done with XSD or DTD schemas, and that is the job of an actual validator rather than a formatter.

The second limitation is that the check runs over tags and not over content. A typo inside the text will not be noticed, because for the formatter text is just characters. What it does catch is nesting such as <b><i></b></i>, and rightly so: elements have to close in reverse order, like a stack.

Namespaces and prefixes

Namespaces exist so that two different vocabularies in one document do not blur into each other. A typical case: an RSS feed uses a namespace for its standard elements, and inside it sits an Atom fragment or a media extension, where elements with the same name from different vocabularies must stay different.

<feed xmlns="http://www.w3.org/2005/Atom"
      xmlns:media="http://search.yahoo.com/mrss/">
  <title>News</title>
  <media:thumbnail url="pic.jpg"/>
</feed>

The xmlns attribute sets the default namespace: every element without a prefix belongs to that vocabulary. The xmlns:media attribute declares the prefix media and binds it to a different address. The address itself never shows up in the data, it works as a label: all that matters is that it matches between the declaration and the use. The rule is simple. If a prefix appears in the document, it has to be declared, otherwise the document is not valid.

The difference with JSON is obvious. JSON has neither prefixes nor namespaces, and the key title always means exactly one thing. In XML the name of an element is a combination of a local name and a namespace, which is exactly why prefixes have to be handled separately during conversion to JSON.

A document or a fragment

A complete XML document starts with an optional but highly desirable XML declaration that states the version and the encoding.

<?xml version="1.0" encoding="UTF-8"?>
<catalog>
  <item id="7">Ropes</item>
</catalog>

The rules are these: a document has exactly one root element, everything else lives inside it, and the declaration comes first with no whitespace and no invisible character in front of it. A fragment cut out of a document has no declaration and may have no root element. Many parsers accept a fragment but will not accept a document without a single root.

Always state the encoding. A file can look perfectly normal in an editor and fall apart during parsing if the declaration says UTF-8 while the bytes are windows-1251. XML 1.0 was published as a W3C Recommendation in 1998, and the requirement to declare the encoding has not changed since.

Turning XML into JSON: the rules and what gets lost

There is no off the shelf conversion from one to the other, so somebody has to agree on the rules. The scheme that works looks like this: the element name becomes a key, the text inside becomes the value, repeated elements are collected into an array, and the attributes move into a separate object under the key @attributes.

<item id="7" stock="3">
  <name>Ropes</name>
  <price currency="RUB">1990</price>
  <tag>tourism</tag>
  <tag>ropework</tag>
</item>

JSON:
{
  "@attributes": { "id": "7", "stock": "3" },
  "name": "Ropes",
  "price": { "@attributes": { "currency": "RUB" }, "#text": "1990" },
  "tag": ["tourism", "ropework"]
}

Notice two things. First, the numbers stay strings, because in XML "1990" is text, and calling it a number requires an external rule such as a schema. Second, a single tag element became a string while two elements with the same name were collected into an array. The structure therefore depends on the data, and whoever consumes the JSON has to handle both cases.

What gets lost during the conversion

• The order of elements, unless it was reflected in the key names. A JSON object is unordered, so a sequence of price first and name second is not preserved anywhere.
• Mixed content: <p>Text <b>in bold</b> continues</p> turns into something vague, because the element now holds a mix of text and a nested node.
• Namespace prefixes, unless they are worked out beforehand. A key like media:thumbnail looks strange in JSON, and it is better replaced with thumbnail plus a note about the source.
• The difference between an element with no attributes and an element with an empty value: <a/> and <a></a> become null and an empty string in JSON, and those are different things.
• Comments and the XML declaration, unless they are pulled out separately.

Order and mixed content are usually left alone when the consumer does not care, or handled by hand. In exports the order of elements often carries meaning, and there you have to agree on a JSON schema in advance instead of relying on automatic parsing.

The other way round: JSON into XML

The reverse conversion is easier, because JSON is built as a tree of keys, and keys turn into tag names very well. The rules mirror each other.

{ "name": "Ropes", "tags": ["tourism", "ropework"] }

<root>
  <name>Ropes</name>
  <tags>tourism</tags>
  <tags>ropework</tags>
</root>

The subtlety is that an array in JSON has no name of its own. While the key tags becomes the root tag everything looks natural, but an array at the top level with no wrapper object has nothing that could hold it. JSON meant for XML is therefore usually wrapped in an object that names the root. One more rule: a key has to be a valid XML name, with no spaces, starting with a letter or an underscore.

The result can be checked in the JSON formatter: it shows an error with a line and a column when the structure is broken, and it builds a tree from the data. More about checking JSON and not breaking it while editing is in the article about checking and formatting JSON.

Common mistakes

• Tags were closed in a different order from the one they were opened in: <a><b></a></b>. They have to close in reverse order, like a stack.
• The ampersand in a link was never escaped: a parameter a=1&b=2 inside an attribute breaks the document, it has to be &amp;.
• XML was copied from an export where the separator was a bare & instead of &amp;.
• It was assumed that JSON allows comments. It does not, it rejects single quotes, and it will not accept a trailing comma.
• It was assumed that numbers stay numbers after the conversion. Without a schema they turn into strings.
• A prefix was declared in one file and forgotten in another. Every XML document stands on its own: a namespace declaration does not carry over from a neighbouring file.

Common questions

What can XML do that JSON cannot?

XML has more machinery: comments, processing instructions, CDATA sections, namespaces, binding to an XSD schema, electronic signatures. That is why XML remains the standard where the format is imposed from outside or where a document has to carry legal weight. But it costs more to parse and the payload is larger, which is why JSON wins inside applications.

The validator says the XML is valid but the program will not read it. What now?

Check three things that real parsers treat more strictly than a simple tag counter. First, the encoding: a declaration of UTF-8 over bytes that are actually windows-1251 gives garbled characters and sometimes a parse error. Second, the XML declaration must have nothing in front of it, no whitespace and no invisible character at the start of the file. Third, the character < must not appear in an attribute value, and the ampersand has to be escaped.

What is CDATA and when is it needed?

It is a section whose content is not parsed as markup: <![CDATA[ any text, even with <b> tags ]]>. People use it when foreign HTML or a piece of code ends up in the data. There are two limits: the sequence ]]> cannot appear inside the section, and a nested section does not work inside CDATA either, because the closing pair has already been used up.

Can XML be compressed?

Yes, and that is a mode of its own. Whitespace between tags and line breaks are removed while the actual text is preserved. Compressed XML is hard for a human to read, so it is used for transfer over a network while the formatted version is kept for reading. The minifier compresses JavaScript, CSS and HTML, and for XML there is a separate button inside the formatter.

What is an XSD, and why have it if there is JSON Schema?

An XSD is a schema describing the structure of an XML document: which elements are allowed, in which order, and with which types. Validation against an XSD gives a strict answer that the document matches expectations, down to values and constraints. JSON Schema does the same job for JSON, and the JSON Schema validator checks your data against a prepared schema. Schemas matter as soon as something other than your own code works with the document.