robots.txt: What It Is, Where It Lives and How to Write It

November 8, 2026

A launch report shows that half the pages of a new site never get indexed even though they sit in the sitemap and return 200. Somebody suggests locking everything down with Disallow, somebody else pastes a password into robots.txt. It gets easier once one fact is clear: robots.txt has nothing to do with protecting the site. Its only job is to tell crawlers which addresses they can skip.

What robots.txt is for

robots.txt is a plain text file at the site root that search crawlers read before they start crawling. It does not decide what ends up in the results, and it does not decide whether a visitor can open a page. It decides only the route and the pace of the crawl.

The file dates back to February 1994, when Martijn Koster at Nexor proposed what was then called RobotsNotWanted.txt and later became known as the Robots Exclusion Protocol. The motivation was practical: one crawler had been too persistent and was loading the server. By June 1994 the format was already a de facto standard, and the protocol itself has barely changed since.

The official document is RFC 9309, «Robots Exclusion Protocol», published in September 2022 as an IETF standard. What matters about it is the scope: it defines the format and how the rules apply, but it does not oblige crawlers to obey. This is a recommendation, not a control mechanism.

it lowers server load, because the crawler does not spend requests on pages that would never make it into the results anyway;
it hides technical addresses from the crawl: carts, site search, sorting pages, the internal API;
it stops duplicate addresses built from query parameters, like a sorted catalogue page;
it is the place where the sitemap address is advertised to crawlers that look for one themselves.

Where the file lives and how it is served

The file has to sit at the site root, at https://example.com/robots.txt. Not in a folder, not with a .html extension, not in a subdirectory. Case does not matter for the lookup itself, but path handling differs across hosts, and the usual explanation for «the file is there but the crawler never sees it» is that it ended up outside the root.

RFC 9309 requires UTF-8 content and the media type text/plain. Line endings may be CR, CR/LF or LF. A strict Content-Type is not mandatory: if the server returns HTML, Google tries to pull the rules out of the response and ignores the rest.

The response code deserves its own paragraph, because the expectations here are often wrong. Under the standard, a 2xx response is parsed and the rules apply. A 4xx other than 429 means the crawler assumes there is no robots.txt and no restrictions at all. A 5xx makes Google pause crawling for 12 hours, then fall back on the last valid copy for up to 30 days if there was one, and only then treat the file as absent. In other words, a robots.txt that returns 500 is worse than having no file at all.

A minimal working robots.txt

User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /search?
Allow: /api/public/

Sitemap: https://example.com/sitemap.xml

To confirm the file is actually served and with which code, the HTTP status reference comes in handy: open the address and look at the status line. A 404 means the crawler reads it as no restrictions at all. A 200 means the contents get parsed. A 301 to the homepage means Google follows up to five hops, and a longer chain behaves like a 404.

What robots.txt does not do

The most common misconception comes from the phrase «closing a page». You cannot close a page through robots.txt, neither for visitors nor for the index. The result is a fairly unpleasant combination: the address is closed off to crawling, the crawler never sees it, and yet if other sites link to it, it can still show up in the results. Without a snippet, because nobody read the content.

it does not restrict access. A page closed with Disallow still opens for anyone who knows the address or receives a link to it;
it does not remove the page from the index. That takes a robots meta tag with a noindex value, or an X-Robots-Tag header;
it does not hide anything from anyone who reads the file directly. Anyone can open https://example.com/robots.txt and look;
it does not replace password protection. If a directory should not be reachable at all, it needs server-side credentials, not a line in a text file;
it does not set the crawl rate for Google: the Crawl-delay directive is not supported by Google at all, while Yandex honours it with a cap.

The practical conclusion: robots.txt answers «which addresses are not worth crawling», not «which addresses to hide». If the task is protection, the answer belongs in server access rules, the territory covered in the full Apache configuration walkthrough.

The directives: User-agent, Disallow, Allow, Sitemap

The file is made of groups. Each group starts with a User-agent line naming the crawler, followed by Disallow and Allow rules. A new User-agent name starts a new group, and the rules before it no longer apply to that group.

User-agent identifies the crawler a block of rules is meant for. An asterisk means every crawler; the name is case-insensitive, unlike the paths;
Disallow is a path that must not be crawled. An empty value, Disallow: with nothing after it, means no restriction;
Allow is a path that may be crawled even when a broader Disallow sits above it;
Sitemap is the full address of a sitemap file. It may appear several times, it is not part of any User-agent group, and it does not follow rule application order;
Crawl-delay is a pause between requests in seconds. Google ignores it completely, Yandex honours it with a limit: anything above two seconds is set through webmaster tools instead.

The order of lines inside a group does not matter. The rule that wins is decided by the length of the matched path, not by the order it was written in. This is the most common reason why Disallow: /admin/ and Allow: /admin/public/ do not behave the way people expect.

How rules apply: the most specific path wins

Under RFC 9309 all matching groups are merged and their rules combined. Every path from Disallow and Allow is matched against the requested address starting from its first byte. Of all the matches, the one containing the most characters wins. If an Allow and a Disallow tie in length, Allow wins. If no rule matches, crawling is allowed.

Opening the public part of a closed directory

User-agent: *
Disallow: /catalog/
Allow: /catalog/brand/

Request /catalog/brand/nike   → 16 characters matched, crawl allowed
Request /catalog/shoes/       → 16 characters matched, crawl blocked

Line order is irrelevant here: the length of the path is what counts.

The comparison runs over the full path including query parameters, and it is case-sensitive. So Disallow: /Search does not cover /search, while Disallow: /search? covers every search result with parameters. To see where the path actually begins inside an address, the URL parser breaks it into its parts.

Wildcards: * and $

Two special characters work inside paths. An asterisk stands for any sequence of characters, and a dollar sign means the match has to end exactly there. Together they let you write a rule for file extensions or for query parameters.

Common patterns

Disallow: /*.pdf$      blocks every PDF in every folder
Disallow: /*?          blocks every address with parameters
Disallow: /*?sort=     blocks sorting only
Disallow: /tmp/*       blocks everything inside tmp
Disallow: /*.php$      leaves /index.php open

The dollar sign is easy to forget. A rule written as Disallow: /*.pdf without it blocks report.pdf and also a page addressed /catalog.pdf-to-png/, because matching happens from the start of the path rather than at its end.

How it ties into sitemap.xml

The Sitemap directive does not affect how access rules apply, but it is how crawlers learn where the sitemap lives. If the file sits anywhere other than the root, a crawler may simply never find it, and a sitemap in a subdirectory only covers the files beneath its own parent directory by default.

Listing the sitemap in robots.txt is useful but not mandatory: a crawler can also look for the conventional /sitemap.xml path, and the most reliable route is adding the address in Search Console as well. The format of the map along with the rules for generating it is covered in the article about sitemap.xml and site maps.

How to check robots.txt

open /robots.txt in a browser and make sure it is plain text, not a placeholder page or a redirect;
check the status code and the content type: you want 200 and text/plain, otherwise the rules may never apply;
confirm the file contains a User-agent group with actual rules, and not just a Sitemap line;
run the contents through a checker: Yandex Webmaster has a robots.txt parser, and the Google Search Console report shows the applied rules and any parse errors;
open a URL you care about and look for contradictions: a page that reads as «indexed» in Search Console while being blocked in the robots.txt report is a mismatch worth fixing.

A separate check concerns characters. The value of a Disallow directive should not carry stray spaces in the wrong place, while a space inside the path itself is perfectly valid and means a literal space. Paths are compared character by character, so a typo in a single letter simply fails to match and the address stays open.

Typical mistakes

Disallow: / on your own site. That is a full crawl block, and pages disappear from the results along with any chance of being seen;
the file placed in a subfolder or named robots.txt.html. A crawler will not find it;
expecting Disallow: /secret/ to protect the contents. It does not, and the page stays reachable through a direct link;
mixing robots.txt with noindex. A page closed to crawling is never read, so the noindex inside it never takes effect;
case confusion. The crawler name is case-insensitive, the path is not;
expecting robots.txt to speed up indexing. The opposite effect is entirely possible: a closed address is not crawled and therefore not evaluated;
a file larger than 500 KiB. Google reads that much and ignores everything beyond it;
duplicate rules for one crawler spread across groups. Under the standard, groups with a matching name are merged, so rules from both of them end up applying.

Common questions

Does a small site need robots.txt?

Strictly speaking, no: with no file at all, a crawler assumes there are no restrictions and walks everything. Even so, a three-line file with a Sitemap directive and a block on the technical directory earns its place on a small project, because it is the simplest way to point the crawler at the map and keep it from wasting effort on administrative URLs.

How do I keep a page out of the index while leaving it reachable?

Use a robots meta tag with a noindex value in the page head, or an X-Robots-Tag: noindex response header. Everybody can still open the page, but the crawler will not include it in the results. Do not do both steps at once: an address blocked in robots.txt is never read, so a noindex inside it goes unnoticed.

Why does a blocked page still show up in the results?

Because that is exactly how it works. Disallow means «do not crawl», not «do not show». If other sites link to the page, the search engine knows it exists and may list the address with an empty description. The only way to remove it from the results is noindex, and only while the page stays reachable to the crawler.

What if robots.txt returns 404?

Nothing needs doing. A 404 response is equivalent to no file as far as the crawler is concerned, and crawling stays unrestricted. If the rules should work, the file has to sit at the root and return 200 with a text/plain content type. The same goes the other way around: if the file exists but the site answers 500, the restrictions do not apply either, and Google additionally pauses crawling for 12 hours.

Can I block the site from a specific crawler?

Yes, with a separate User-agent group naming that crawler, for example User-agent: GPTBot. The syntax is the same as for the shared group. Keep in mind that this is a request rather than a ban: a crawler that ignores the protocol will not respect a Disallow either. Real protection requires server-side access control.

How do I see which rules were applied?

Search Console has a robots.txt report that shows which lines were parsed, which were ignored and which addresses are blocked. Yandex offers the equivalent in its webmaster tools. Manual checking works too: take a URL, pull it apart with the URL parser, and compare the path against the rules, remembering that the longest matching line is the one that counts.

The rule that settles most cases: block only what genuinely does not need crawling, leave everything else open, and point crawlers at it through Sitemap. A file that blocks useful pages does more harm than no file at all.