Select which bots these rules apply to, then add Allow/Disallow paths.
Robots.txt Generator creates properly formatted robots.txt files for controlling search engine crawler access to your website. Select which crawlers to target (all bots with User-agent: *, or specific crawlers like Googlebot, Bingbot, or GPTBot), choose which paths to allow or disallow, and set a crawl delay. The generator validates the syntax and shows a preview of what each crawler can and cannot access.
The robots.txt file (placed at the root of a website, e.g., https://example.com/robots.txt) follows the Robots Exclusion Protocol — a convention that well-behaved crawlers honor to respect a site's crawl rules. Key directives: User-agent (which bot the rules apply to), Disallow (paths to block), Allow (paths to explicitly allow within a broader Disallow), Crawl-delay (seconds between requests), and Sitemap (URL of the XML sitemap).
Robots.txt controls crawlers, not page indexing. A page disallowed in robots.txt can still appear in search results if another page links to it. To prevent indexing, use the noindex meta tag or X-Robots-Tag response header. Common robots.txt blocks: admin panels (/admin/), API endpoints (/api/), staging environments, internal search results pages (/search?), print views, and AI training crawlers (GPTBot, CCBot, Common Crawl).
Block admin area from all crawlers
Result: User-agent: * / Disallow: /admin/ / Disallow: /api/ / Sitemap: https://example.com/sitemap.xml
Block AI training crawlers
Result: User-agent: GPTBot / Disallow: / / User-agent: CCBot / Disallow: / (blocks OpenAI and Common Crawl)
Allow Googlebot but block others
Result: User-agent: Googlebot / Allow: / / User-agent: * / Disallow: /
Does robots.txt guarantee pages won't be crawled?
No — robots.txt is a convention that well-behaved crawlers (Google, Bing, legitimate search engines) follow voluntarily. Malicious bots and scrapers ignore it completely. It does not provide security — never rely on robots.txt to protect sensitive content. For actual access control, use server-side authentication, IP allowlisting, or HTTP Basic Auth. Google and Bing do honor robots.txt; links to disallowed pages can still appear in search results without content.
What is the difference between robots.txt and meta robots?
robots.txt: prevents crawling — the bot never downloads the page. Applies site-wide. Takes effect immediately but crawl rate changes may take time. meta robots: prevents indexing/following of a page the bot has already downloaded. Added to the
of each page. More granular control per page. If you need to de-index a page already in Google's index, add — robots.txt cannot remove already-indexed pages. For removing specific URLs from search results, use Google Search Console's URL Removal tool.How do I block AI training crawlers?
Several AI companies have published their crawler User-agents: OpenAI (GPTBot), Anthropic (ClaudeBot), Google AI (Google-Extended), Meta AI (Meta-ExternalAgent), Common Crawl (CCBot), Amazon (Amazonbot). Add User-agent: GPTBot / Disallow: / for each you want to block. Note: many AI companies trained models before these crawlers existed. These rules only affect future crawls. AI.txt (an emerging standard similar to robots.txt but for AI) is being developed but not yet widely adopted.
What is the Sitemap directive in robots.txt?
The Sitemap directive in robots.txt tells crawlers the URL of your XML sitemap: Sitemap: https://example.com/sitemap.xml. This is not part of the official robots.txt spec but is de-facto standard and supported by all major search engines. Multiple Sitemap directives are allowed. The Sitemap directive is outside any User-agent block — it applies globally. You can also submit sitemaps directly in Google Search Console and Bing Webmaster Tools, but robots.txt discovery is useful for any crawler.
What is crawl budget and how does robots.txt help?
Crawl budget is the number of URLs Googlebot crawls on your site in a given time period, determined by your server's capacity and Google's crawl rate limits. For large sites (millions of pages), wasting crawl budget on low-value pages (parameters, facets, print versions) means important pages get crawled less frequently. Disallowing low-value paths in robots.txt frees crawl budget for important content. For small sites (under 1,000 pages), crawl budget is rarely a concern — Google crawls small sites fully regardless.