Robots.txt Generator
Robots.txt generator: assemble Disallow, Allow, Clean-param, Host and Sitemap rules for Yandex and Google, then download a ready robots.txt file.
Why a site needs a robots.txt file
A robots.txt file is a plain text file in the root of a website that tells search crawlers which addresses to visit and which to skip. The question of what robots.txt is and how to create one comes up with every launch and every domain migration, because the syntax looks heavier than it is: there are only a handful of directives to learn. This robots.txt generator turns them into checkboxes. Block a private folder, block an admin panel, block JSON files and pages carrying query parameters, keep the admin folder open for bots, add the Yandex-only Clean-param and Host lines, and point the Sitemap directive at your sitemap file. Everything is assembled in the browser, nothing is uploaded anywhere, and the finished text is ready to copy or download.
What the robots.txt generator does
- Six ready-made crawl rules for blocked folders, JSON files, session parameters and admin areas
- A custom path field that appends any Disallow rule to the bot group
- A choice of recipient: all bots, Yandex, Googlebot, Bingbot or a bot you type in
- A service block for Clean-param, Host and Sitemap with the domain filled in for you
- The finished text with explanatory comments, plus copy and download buttons
- A reminder about status 200, the text/plain content type and the file location in the root
How to create a robots.txt file
- Type your domain and check the file path, which defaults to /robots.txt
- Tick the crawl rules you need in the checklist
- Pick who the file is addressed to: every bot or one specific crawler
- Add a custom Disallow path and the service lines if you need them
- Copy the text or download the file and place it in the site root
How the rules are assembled
The file is built as a list of lines. First comes a commented header explaining where the file belongs, which status code it must return and why a rule behaves like a prefix. Next comes the User-agent line with the selected bot name, followed by the active Disallow and Allow directives in the order they were listed in the interface. The Yandex-only directives Clean-param and Host are appended to the end of the same group, while the Sitemap line is pushed outside the group because the protocol does not tie it to a particular crawler. Empty input in the Disallow field is skipped, and stray spaces and trailing slashes are normalised so the file stays tidy when it is pasted around.
robots.txt rules and what each one does
| Rule | What it blocks | Comment |
|---|---|---|
| Disallow: / | The whole site | A last resort, mostly for staging |
| Disallow: /private/ | Accounts, drafts and private areas | The rule behaves like a prefix |
| Disallow: /admin/ | The control panel | Individual paths can still be opened with Allow |
| Disallow: /*.json$ | Every JSON file | The dollar sign anchors the end of the path |
| Disallow: /*?session= | Addresses carrying a session parameter | More precise than the blanket parameter block |
| Disallow: /*?* | All pages with query parameters | Useful against endless duplicate URLs |
| Allow: /wp-admin/ | Opens the admin folder | Works together with Disallow: / |
| Clean-param: ref /page | A parameter that carries no content | Yandex directive, ignored by Google |
| Host: example.com | The primary version of the domain | Yandex directive, ignored by Google |
| Sitemap: https://example.com/sitemap.xml | The address of the sitemap | Standard directive, read by everyone |
| User-agent: Yandex | A separate group of rules | Lets you split the rules per crawler |
| Disallow: | Nothing | An empty value cancels a broader rule |
These lines are exactly what the generator produces, and the finished file comments each of them in place.
Where a robots.txt file really earns its place
The first and most common case is an online shop. The basket, the customer account and the checkout page hold nothing worth showing in a search result, yet all of those addresses index happily unless you close them in advance. A result page for a running shoe query should never have surfaced a promo code input. The second case is a forum or a blog with comments: site search, tag archives, sorting pages and pagination all produce dozens of nearly identical addresses, and without a robots.txt file they end up in the index. The list of sections to close is tidied up in advance, and sorting lines shows the duplicates and the inconsistencies that would later be copied into the rules. The third case is a site that is already open on its domain but is not ready to be discussed with search engines yet.
A separate category is made up of sites with a login area. Sometimes a robots.txt file blocks such an area out of the box, and that is a mistake: a closed section still lands in the index as a page without a description, and a search for the site operator with a site: query finds it easily. A fourth case is a migration to a new domain, where the old robots.txt holds rules for the old structure and has to be rewritten, otherwise new sections get closed by accident. The fifth case is large catalogues that keep hundreds of thousands of addresses in the index, where rules help take everything that is not a product out of the crawl.
- Online shops: basket, account and checkout pages
- Forums and blogs: search, tags, pagination and archives
- Sites with a login area behind authentication
- Staging and test builds on the main domain
- Migrations to a new domain with a changed structure
- Catalogues with a large number of service addresses
Common mistakes in robots.txt
The first mistake is placing the file somewhere other than the root. Put robots.txt inside a site folder or a static subfolder and no robot will find it, so every rule quietly stops working. The second mistake is blocking pages that should be indexed: closing the whole site with Disallow: / and forgetting about it. The third is blocking a page and tagging it noindex at the same time. The robot never loads the page, so it never sees the tag, and what remains in the index is a bare address with an empty description. Looking at the page itself helps while checking this, because the page preview shows what a crawler would actually receive.
The fourth mistake is assuming robots.txt protects private data. It does not: a blocked page is still reachable by anyone who types the address into a browser. If data should not be visible at all, that calls for authentication, not a rule for crawlers. The fifth is confusing Allow and Disallow: an Allow rule does not override a more specific Disallow, so blocking /admin/ first and then allowing /admin/login changes nothing. The sixth is being too broad with parameter rules: Disallow: /*?* also closes off perfectly reasonable addresses that carry campaign tracking parameters, and those are the links your advertising sends people through.
- The file sits in a folder instead of the site root
- Disallow: / closes the whole site and nobody notices
- A page is blocked from crawling and from indexing at once
- robots.txt is treated as a way to hide private data
- Allow: /admin/login fails on top of Disallow: /admin/
- Disallow: /*?* also cuts off legitimate tagged URLs
A step by step walkthrough of the file
Take a typical online shop on the domain example.com. The task: close the customer account and the control panel, leave the WordPress admin folder reachable for bots, block JSON exports, declare the primary domain and wire up the sitemap. Step one is the domain: write example.com with no protocol, so that the Host and Sitemap directives do not end up with https:// twice. Step two is the rules: tick the blocks for /private/ and /admin/, block *.json and enable the /wp-admin/ exception. Step three is the recipient: keep the asterisk so the file applies to every crawler. The address of the sitemap itself is needed next, and the sitemap generator assembles it separately.
What follows is the file itself. It opens with a block of comments explaining that the file has to live at https://example.com/robots.txt and be served with status 200 and the type text/plain. Then comes User-agent: * and the rules. Then the Yandex service lines: Clean-param with the ref parameter and Host with the domain. The file closes with the Sitemap line pointing at the full address of the sitemap. That order is not mandatory, but it is familiar: crawlers read the file line by line, and people read it from the top, seeing the explanation before the rules.
What happens at each step of the build
| Step | What it does | What lands in the file |
|---|---|---|
| 1 | Header | Writes the domain and the 200 warning |
| 2 | Bot choice | Decides who the group is for |
| 3 | Blocks | Adds the active Disallow and Allow rules |
| 4 | Custom path | Appends any path you typed |
| 5 | Clean-param | Adds a parameter that should not create duplicates |
| 6 | Host | Fixes the primary version of the domain |
| 7 | Sitemap | Substitutes the domain into the template |
| 8 | Checks | Counts the rules and characters, shows the result |
The file is assembled entirely in the tab: not a single character of robots.txt reaches a server until you copy or download it yourself.
robots.txt, sitemaps and the rest of the picture
The files that usually get mentioned together do different jobs, and mixing them up is easy. robots.txt tells a crawler where not to go. A sitemap tells it where to go and which addresses count as the important ones. A sitemap is a list, not a ban, so an address closed in robots.txt has to be removed from the sitemap, otherwise you end up with a contradiction that search engines resolve in unpredictable ways. For hands-on work with the file list itself, the sitemap generator shows what would actually reach the index, while robots.txt decides what will not.
A second cluster is tags and page headers. Title and description are collected by the meta tag generator, and robots.txt has no say in them: it manages crawling, not headings. The third element is the .htaccess file, which handles redirects and response codes at the server level, and its rules are written with the .htaccess generator. The fourth is technical headers such as X-Robots-Tag, which the server sends along with the page itself. If you are new to any of this, start simple: one file in the root, one Sitemap line, and no blanket blocks at all. It is worth checking separately that the server returned the file as plain text, and the HTTP status reference explains what the response code means.
- robots.txt manages crawling, a sitemap lists the addresses
- An address closed in robots.txt should not appear in the sitemap
- Title and description live in meta tags, not in robots.txt
- Response codes and redirects are a server concern, not a robots.txt one
- A noindex tag only counts if the robot can reach the page
- An empty robots.txt with a Sitemap line is already a useful file
Common questions about robots.txt
What is robots.txt and where does the file go
How do I create a robots.txt file for my site
Does Disallow match exactly or act as a prefix
What does Allow do and how do I open the admin folder
How do Clean-param and Host differ from ordinary rules
What happens if robots.txt returns a 404 or a 500
Does blocking a page in robots.txt stop it from being indexed
Is a robots.txt file worth writing if the site works fine
Related tools
Sitemap.xml Generator
Build a valid sitemap.xml from a URL list or a CSV: lastmod, changefreq, priority, limit checks and a downloadable file
Favicon Generator
Make a site icon from a monogram or your own picture: favicon.ico, apple-touch-icon, Android icons, site.webmanifest and ready HTML tags
JSON Schema Generator
Build a draft JSON Schema from a JSON sample: field types, required keys, nested objects and arrays
Regex Generator
Build a regular expression from a task template or break your own pattern down into parts
Open Graph Generator
Build og:title, og:description, og:image and Twitter Card tags for the link preview on social platforms
ASCII Table
Complete ASCII reference: codes 0–127 in DEC, HEX, OCT and BIN with search by character or code
Regex Tester & Debugger
Test regular expressions with match highlighting, groups, and common patterns
HTML Meta Tag Generator
Generate SEO-optimized HTML meta tags for your website