Sitemap.xml Generator

When a sitemap actually pays off

The first case is a new site. Crawlers learn about pages through links, and a fresh domain has no links yet, so the sitemap turns into the list the crawl starts from, working outwards from the home page and the sections. The second case is a large catalogue where crawling every product takes days and half the links lead nowhere. Here the map saves crawler time: it follows a list instead of grinding through endless pagination. The third is a site with poor internal linking, where important pages sit three clicks deep and are barely mentioned in the menu. The fourth is a site that has just moved, where old addresses are closed and the new ones have not started working yet.

The fifth case is an online shop where every product has several addresses: with a colour, with a size, with a filter. The map helps the crawler see which of those variants count as primary and which are merely variations of one product. The sixth is a site with non-Latin addresses, where some pages are reachable by transliteration and some in the original script. The seventh is any site where freshness matters: a news section publishing every hour, or a catalogue whose stock changes constantly. The eighth is a resource with a big archive, where old pages have not been touched for years and the map helps the crawler tell fresh content from archived content.

• A new site with no inbound links yet
• A large catalogue where link crawling cannot keep up
• Important pages that are barely linked from anywhere
• A migration to a new domain with changed addresses
• A shop with filters, colours and product variants
• A site with transliterated addresses and Latin duplicates

Frequent mistakes when building a sitemap

The first mistake is putting everything into the map. Service addresses, search pages, filtered listings and duplicates all look like ordinary pages, but for the crawler they are noise that stretches the crawl. The second is including campaign tags: tracking parameters have no business in a sitemap, and the crawler normalises addresses to their canonical form anyway. The third is forgetting the protocol and writing addresses as /about. Such a file is formally invalid, and some search engines parse it only partially or not at all.

The fourth mistake is putting a date in the future, or refreshing it without any change. Crawlers notice quickly that lastmod moves while the content does not, and stop trusting the field. The fifth is duplicates: the same address written once with www and once without, or with a trailing slash and without, counts as two lines, though it is one document for a search engine, and the repetition lowers trust in the whole file. The sixth is uploading the file anywhere other than the root. robots.txt only works from the root, and the same applies to a sitemap referenced through a Sitemap directive.

• Service pages and internal search results end up in the map
• Addresses are listed with advertising tracking parameters
• Addresses without a protocol: /about instead of a full URL
• lastmod changes although the page itself did not
• The same address is written several times
• The file is uploaded somewhere other than the site root

A step by step walkthrough

Take a blog on the domain example.com with 24 pages. The task: build the map from a CSV export, set the change dates and keep service addresses out. Step one is the domain: type example.com so that relative addresses become absolute. Step two is the source: choose CSV, because the addresses arrived in a spreadsheet where the first column holds links and the second holds the edit date. Step three is pasting the export: the url,lastmod header is detected and skipped, and lines like https://example.com/posts/...,2026-02-11 are split into an address and a date. When the export arrives as plain text rather than CSV, it is first turned into a table, for instance with the CSV to JSON converter.

Once the list is parsed, the file is assembled. Each address gets a url block with four lines: loc with the full address, lastmod with the date from the CSV or with the common date, changefreq with the chosen frequency and priority with the chosen number. All elements are written in the order the protocol expects: loc first, then lastmod, changefreq and priority. The urlset root with the namespace is written once at the top and closed at the end. What remains is to check the counters: how many addresses made it into the file, how many duplicates were removed, and whether the weight fits inside 50 MB. If the file refuses to parse, the XML formatter usually points at the broken tag.

Everything happens in the memory of the tab: neither the address list nor the finished XML leaves your computer until you copy it yourself.

Sitemaps, robots.txt and meta tags: who does what

These three are often mixed up, because all of them concern search and all of them live in text files. A sitemap is the list of addresses worth visiting and worth treating as primary. robots.txt is a set of crawl restrictions, and those rules are assembled by the robots.txt generator. Meta tags such as title and description are what the robot sees once a page makes it into the results, and they are put together by the meta tag generator. The connection reads like this: the sitemap says where to go, robots.txt says where not to go, and the meta tags decide how the page will look in the results.

There is one more layer, the .htaccess file, where the server defines redirects and response codes including X-Robots-Tag; its rules are written with the .htaccess generator. A word on verification: first make sure the sitemap is reachable and served as XML, then submit it in the webmaster panel and wait for it to be processed, and only then judge it by the indexing figures. Otherwise it is easy to lose a day looking at numbers that have not been calculated yet. The response code of the map itself deserves a separate check, and the HTTP status reference tells a working file from an error page.

• A sitemap lists addresses, robots.txt forbids crawling
• Title and description decide how the page looks in the results
• Response codes and redirects live in the server configuration
• A page closed in robots.txt should not stay in the sitemap
• Check that the file is reachable before reading the statistics
• Large sites build a sitemap index instead of one giant file