Set Allow and Disallow rules for Googlebot, Bingbot, and AI crawlers, add your sitemap, and check any URL against the result. The file updates as you type.
Start from a preset
Rules
User-agent
Disallow (paths to block)
Allow (exceptions to a Disallow)
Crawl-delay (optional, seconds)
Block AI crawlers
Each crawler you select gets its own rule with "Disallow: /"
Sitemap
None added (optional)
Test a URL
Checks a path against the rules above. The longest matching rule wins, and Allow wins a tie.
Crawler
URL or path
Allowed
No Allow or Disallow rule matches this path (User-agent: *).
robots.txt preview
User-agent: * Disallow:
Check before you upload
• No Sitemap added. If you have an XML sitemap, add its full URL so crawlers can find it.
Build a robots.txt File for Search Engines and AI Crawlers
A robots.txt file tells crawlers such as Googlebot, Bingbot, GPTBot, and ClaudeBot which parts of your site they may request. Crawlers check it before they fetch your pages, so it is the first place to set crawl rules.
Pick a user-agent, then add Disallow paths to block and Allow paths for exceptions. You can set a Crawl-delay, add one or more Sitemap URLs, and tick AI crawlers to give each one its own Disallow: / rule. The file in the preview updates as you type, and you can copy it or download it as robots.txt.
Presets cover the common starting points: allow all, block all, a WordPress setup, and blocking AI bots. Before you upload, use the URL tester to see whether a given crawler can reach a given path, and read the warnings for common mistakes.
Common Reasons to Edit Your Crawl Rules
Launching or redesigning a site
Set up rules for admin, login, and cart paths before launch, or rewrite them after a CMS or theme change.
Opting out of AI training crawlers
Block GPTBot, ClaudeBot, CCBot, and other AI crawlers while leaving Googlebot and Bingbot free to crawl.
Trimming duplicate and low-value URLs
Disallow internal search results, filter and sort parameters, and session URLs so crawlers spend time on pages you want indexed.
Pointing crawlers to your sitemap
Add your XML sitemap as a full URL in the correct Sitemap: format, with one line per sitemap.
Create and Upload the File in 4 Steps
- Choose a preset (Allow all, Block all, WordPress, or Block AI bots), or start from the empty default rule.
- Click Add rule to pick a user-agent, then add paths to Disallow and any exceptions to Allow.
- Tick the AI crawlers you want to block, and add your sitemap URL.
- Use Test a URL to check key pages, then copy the file or download robots.txt and upload it to your site's root.
Note: Everything runs in your browser. Nothing you enter is uploaded.
Limits to Know Before You Publish
- Robots.txt is a request, not access control. Well-behaved crawlers follow it, but some scrapers ignore it. Protect private pages with a login, not a Disallow line.
- A blocked URL can still show up in search results if other sites link to it. Use noindex to keep a page out of results.
- The file only works at the root of a host, such as https://example.com/robots.txt. Each subdomain needs its own file.
- The URL tester follows Google's documented matching. Other crawlers may read edge cases differently.
User-Agent Names for Googlebot, Bingbot, GPTBot, and Other Crawlers
Common user-agent names you can use in the User-agent line.
| User-agent | Type | What it is |
|---|---|---|
| * | All crawlers | Wildcard that applies to every crawler without its own rule |
| Googlebot | Search engine | Google's main web search crawler |
| Googlebot-Image | Search engine | Crawler for Google Images |
| Bingbot | Search engine | Microsoft Bing's crawler |
| GPTBot | AI crawler | OpenAI's crawler for collecting AI training data |
| ChatGPT-User | AI crawler | Fetches pages when a ChatGPT user asks it to browse |
| Google-Extended | AI opt-out token | Controls whether Google uses your content for its AI models such as Gemini |
| ClaudeBot | AI crawler | Anthropic's crawler for collecting AI training data |
| CCBot | AI crawler | Common Crawl's crawler, whose archive many AI models use |
| PerplexityBot | AI crawler | Perplexity's search crawler |
| Bytespider | AI crawler | ByteDance's crawler |
Crawler names and behavior can change. Check each company's official documentation for the current user-agent names.
Allow vs. Disallow Syntax, Wildcards, and Rule Priority
How the directives work together, and where people usually trip up.
Disallow asks crawlers to stay away; it does not hide a page
Disallow is a request that compliant crawlers follow. It does not block access, and it does not remove a URL that is already indexed.
Use it to manage crawling. Use noindex or a login to control what appears in search or who can see a page.
Which rule wins: Allow or Disallow
When rules in the same group conflict, Google applies the one with the longest matching path. With Disallow: /blog/ and Allow: /blog/public/, pages under /blog/public/ stay crawlable. If the two are the same length, Allow wins.
A crawler that has its own group follows only that group and skips the * group, so repeat shared paths in each group.
Wildcards and end anchors: * and $
Paths match as prefixes, so Disallow: /blog also matches /blog-news. Google and Bing support * to match any characters and $ to mark the end of a URL. For example, Disallow: /*.pdf$ blocks URLs ending in .pdf, and Disallow: /*?sessionid= blocks URLs with that parameter.
Writing the Sitemap line correctly
Use a full URL, for example Sitemap: https://example.com/sitemap.xml. A relative path such as /sitemap.xml is not valid. You can list more than one Sitemap line, and it does not need to sit inside a user-agent group.
Mistakes That Block Your Whole Site (and How to Fix Them)
Leaving Disallow: / on after launch
A staging site often uses User-agent: * with Disallow: /. If that file is copied to production, crawlers stop fetching the whole site.
💡 See the solution▼
Listing secret URLs in a public file
Anyone can open /robots.txt. Adding /admin/ or a private folder to it tells everyone where those paths are.
💡 See the solution▼
Missing the trailing slash
Disallow: /blog matches /blog, /blog/, and also /blog-news and /blogger. Disallow: /blog/ matches only paths inside the folder.
💡 See the solution▼
Using Disallow when you need noindex
If you block a page with Disallow, Google can't open it to read a noindex tag. The URL may stay in results with no description.
💡 See the solution▼
Adding a crawler rule that overrides your * rules
You block /private/ for User-agent: *, then add a Googlebot rule for something else. Googlebot now follows only its own rule and can crawl /private/.
💡 See the solution▼
FAQ: Blocking Pages, Crawl-delay, and Where to Upload
Q.What is a robots.txt file and do I need one?
A. A robots.txt file is a plain text file that tells crawlers which parts of your site they may request. It sits at the root of your domain, for example https://example.com/robots.txt. You don't need one for a small site where everything can be crawled, but it helps when you want to keep crawlers out of admin pages, internal search results, or duplicate URLs, or opt out of specific AI crawlers.
Q.Will Disallow remove a page from Google?
A. No. Disallow stops compliant crawlers from fetching the page, but the URL can still appear in search results if other sites link to it. To keep a page out of search results, use a noindex meta tag or an X-Robots-Tag header, and leave the page crawlable so Google can see that instruction.
Q.How do I block ChatGPT, Claude, and other AI crawlers?
A. Add a separate rule for each crawler's user-agent with Disallow: /. For example, User-agent: GPTBot followed by Disallow: / blocks OpenAI's training crawler. Use the Block AI crawlers panel to add GPTBot, ClaudeBot, CCBot, and others in one click. Blocking works only for crawlers that follow robots.txt, and it may also keep your content out of AI search and answer features.
Q.Where do I upload the robots.txt file?
A. Upload it to the root of the host, so it loads at https://yourdomain.com/robots.txt. A file in a subfolder is ignored. Each subdomain, such as blog.example.com, needs its own file at its own root, while one file covers every folder on the same host.
Q.Does Google support Crawl-delay?
A. No. Google ignores the Crawl-delay line. Some other crawlers, such as Bing and Yandex, honor it. This tool lets you add it per user-agent and warns you when it is set.
Q.Can I allow one page inside a blocked folder?
A. Yes. Add a Disallow for the folder and an Allow for the specific path. When both match, Google applies the longer, more specific path, and if they are the same length Allow wins. For example, Disallow: /wp-admin/ with Allow: /wp-admin/admin-ajax.php keeps that one file crawlable.
Q.Is there a size limit for robots.txt?
A. Google reads up to 500 KiB of a robots.txt file and ignores anything past that. Files made with this tool are far smaller, but if you maintain a very long file by hand, combine rules with wildcards where you can.
Q.Is anything I type sent to a server?
A. No. The file is built in your browser as you type. Your domain, paths, and sitemap URLs are not uploaded or saved, and they are gone when you close or reload the page.
Your feedback
makes this tool better.
Tell us which features you'd like, what should be fixed, or which new tool you'd like to see. An engineer reads every submission and uses it to guide development.