Robots.txt Generator and Tester
Build a robots.txt in the generator, or paste yours into the tester, enter a crawler and a URL path, and see whether it is allowed or blocked, with the deciding rule highlighted. The tester follows RFC 9309: the most specific User-agent group wins, then the longest matching rule, and Allow wins a tie. For the example file, Allowed: ExampleBot may crawl /private/press/kit.html. Allow: /private/press/ (line 4, 15 characters) beats Disallow: /private/ (line 3, 9 characters), because the longest matching rule wins. Wildcards (*, $) and case-sensitive paths are handled exactly as the standard describes. Nothing you paste leaves your browser.
Updated Runs in your browser: nothing you enter is uploadedHow this tool works
Robots.txt generator
Stays in your browserPick a default, add rules (one path per line) and crawler-specific groups. The file updates as you type.
Your robots.txt
Adjusted or left out
- robots.txt is public and only a request to well-behaved crawlers. Do not use it to hide private pages.
Robots.txt tester
Paste a robots.txt, then enter a crawler and a URL (or just its path). The rules are applied as RFC 9309 describes.
A product token such as Googlebot or bingbot, * for any other crawler, or a full User-Agent string.
Allowed
Allowed: ExampleBot may crawl /private/press/kit.html. Allow: /private/press/ (line 4, 15 characters) beats Disallow: /private/ (line 3, 9 characters), because the longest matching rule wins.
No group names ExampleBot, so the * group applies (line 2).
The file, with the deciding rule highlighted
- 1
# Example robots.txt - 2
User-agent: *In the group used - 3
Disallow: /private/Matches, but loses - 4
Allow: /private/press/Deciding rule - 5
Disallow: /*.pdf$In the group used - 6
- 7
User-agent: Googlebot - 8
Disallow: /nogoogle/ - 9
- 10
Sitemap: https://example.com/sitemap.xml
Lines the tester ignored
How path patterns match
| Pattern | Matches | Does not match | Rule |
|---|---|---|---|
| /fish | /fish, /fish.html, /fishheads/yummy.html | /catfish, /Fish.asp, /?id=fish | A rule is a prefix: it matches every path that starts with it, and case counts. |
| /fish/ | /fish/, /fish/salmon.htm | /fish, /fish.html | A trailing slash limits the rule to that folder and what is inside it. |
| /*.php | /filename.php, /folder/filename.php?parameters, /folder/any.php.file.html | /, /windows.PHP | “*” matches any run of characters, including none. |
| /*.php$ | /filename.php, /folder/filename.php | /filename.php?parameters, /filename.php5 | “$” at the end means the path must end exactly there (the query string is part of the path). |
| /fish*.php | /fish.php, /fishheads/catfish.php?parameters | /Fish.PHP | Wildcards can sit in the middle. |
Who may fetch what: the RFC 9309 simple example
| Crawler | /example/page.html | /example/other.html | /publications/a.html | /images/pic.gif |
|---|---|---|---|---|
| otherbot | Blocked | Blocked | Allowed | Blocked |
| foobot | Allowed | Blocked | Blocked | Blocked |
| barbot | Blocked | Allowed | Allowed | Allowed |
| quxbot | Allowed | Allowed | Allowed | Allowed |
Directives and who reads them
| Line | What it does | |
|---|---|---|
| User-agent: name | Starts a group for the crawler with that product token (or * for all others). Consecutive User-agent lines share the rules that follow. | Supported |
| Disallow: /path | Asks matching crawlers not to fetch paths that match. An empty value blocks nothing. | Supported |
| Allow: /path | Makes an exception to a broader Disallow. The longest matching rule wins; Allow wins a tie. | Supported |
| Sitemap: URL | Gives the full address of a sitemap. It is not part of any group and can appear anywhere. | Supported |
| Crawl-delay: seconds | Asks a crawler to wait between requests. Not part of RFC 9309. | Ignored (Bing and some others read it) |
How to create and test a robots.txt file
- Choose the default. In the generator, pick “Allow everything” (an empty Disallow) or “Block everything” (Disallow: /) for all crawlers, then list the exceptions.
- Add rules per crawler. Write one path per line under Disallow or Allow. Add a group for a specific crawler such as Googlebot or GPTBot; it will ignore the * group completely.
- Add your sitemaps. List each sitemap as a full https:// address, one per line.
- Copy or download. Copy the file or download it as robots.txt and upload it to the root of the site, so it is reachable at /robots.txt.
- Test a URL. Send the file to the tester (or paste any robots.txt), enter a crawler and a path, and read the verdict. The deciding line is highlighted, and so are the rules that matched but lost.
Frequently asked questions
- Which rule wins when Allow and Disallow both match?
- The one with the longer path pattern, counted in characters including any * and $. If both are the same length, Allow wins. The order of the lines does not matter. RFC 9309 calls this the most specific match; Google documents the same behaviour.
- How does a crawler pick its group?
- It looks for groups whose User-agent matches its product token, ignoring case, and uses only those (several groups for the same crawler are merged). Only when none matches does it use the * group. It never combines a specific group with *, so a Googlebot group must repeat any * rule you still want applied to Googlebot.
- Are paths case-sensitive?
- Yes. /Private/ and /private/ are different paths. User-agent names and field names (Disallow, Allow) are not case-sensitive.
- What do * and $ do?
- “*” matches any run of characters (including none) and “$” at the end of a pattern means the URL must end there. So Disallow: /*.pdf$ blocks every URL ending in .pdf but not /file.pdf?download=1. Rules are otherwise prefixes: Disallow: /fish also blocks /fish.html and /fishheads.
- Does Google honor Crawl-delay?
- No. Google ignores Crawl-delay and manages its own crawl rate. Bing and some other crawlers do read it, which is why the generator can add it, but it is not part of RFC 9309.
- Does Disallow keep a page out of Google?
- Not reliably. It stops crawling, but a blocked URL that other pages link to can still be indexed, without a description. To keep a page out of search results, let it be crawled and use a noindex robots meta tag or an X-Robots-Tag header, or put it behind a login.
- What happens with an empty file, a missing file or an error?
- An empty robots.txt, or one with no group for the crawler, allows everything. Under RFC 9309 a missing file (a 4xx response) also means no restrictions, while a server error (5xx) means the crawler should assume everything is blocked. This tool tests the text you paste; it does not fetch your site.
- How big can robots.txt be?
- Crawlers must read at least the first 500 KiB (512,000 bytes); Google ignores anything after that. The tester stops reading at the same point and tells you if your file is longer.
- How are percent-encoded and non-English paths compared?
- Non-ASCII characters are converted to their UTF-8 percent-encoded form, and encoded unreserved letters and digits are decoded, before comparing. So /foo/bar/ü matches /foo/bar/%C3%BC, and /foo/bar/%62%61%7A matches /foo/bar/baz. An encoded slash (%2F) is not the same as a slash.
- Is my robots.txt sent anywhere?
- No. Generating and testing run in your browser; the text, paths and crawler names are never uploaded or stored. The only thing the site records, if you allow analytics, is that a button was used.
How it works
The tester implements RFC 9309 (Robots Exclusion Protocol, September 2022). Lines are read as “field: value”; field names are case-insensitive; “#” starts a comment; a leading byte order mark is skipped; lines may end in LF, CR or CRLF; and only the first 500 KiB is read. Consecutive User-agent lines open one group. The crawler’s product token (the name you enter, or the name after “compatible;” if you paste a full User-Agent string) is compared with each User-agent value, ignoring case; all groups that name it are merged, and if none does, the groups for * are merged. A crawler name is matched whole, so a rule for “googlebot” does not apply to “googlebot-image”; Google’s own crawlers have additional fallbacks described in its documentation that are not part of the RFC. A rule matches when its path pattern is a prefix of the URL’s path plus query. Both are percent-encoded to ASCII (UTF-8) and unreserved characters are decoded before comparing; “*” matches any run of characters and a final “$” anchors the end; comparison is case-sensitive. Of the matching rules the one with the longest pattern wins (length counted after normalisation, including * and $), and Allow wins a tie. A group with no matching rule, or no group at all, allows the URL. A path that does not start with “/” is ignored, as is a rule before any User-agent line. The generator writes the same syntax: one group per crawler, Disallow lines then Allow lines, Crawl-delay if you ask for it, and Sitemap lines at the end.
Sources
More free tools
- URL ParserSplit a URL into protocol, host (punycode and Unicode), port, path and an editable query table. Strip tracking parameters and resolve relative links.
- URL Encode & DecodePercent-encode or decode URLs and query strings with correct UTF-8, split a URL into parts and parameters, and catch double-encoding. Runs in your browser.
- HTTP Status Code LookupLook up any HTTP status code or phrase: its meaning, class, defining RFC section, default cacheability, and common causes and fixes. Filterable IANA table.
- URL Slug GeneratorTurn titles into clean URL slugs: hyphen or underscore, accents to ASCII, optional stop-word removal and a max length. Batch mode warns about duplicates.
- User Agent ParserPaste a User-Agent string to see the browser and version, rendering engine, OS, device type and notable tokens. Explains UA reduction. Runs locally.
- Meta Tag GeneratorGenerate a copy-ready head snippet: title, description, canonical, robots, Open Graph and X card tags, with length checks and search and social previews.
More: Web, URLs, images and networkAll developer toolsAll toolsHome & DIY calculators