A robots.txt file is a plain text file at the root of your domain that tells crawlers which URLs they may request. The simplest valid example, which allows everything, is a User-agent: * line followed by an empty Disallow: line. Below you will find copy-ready files for the most common setups, an AI crawler section, and the steps to test your file before it goes live.
One rule before the examples: robots.txt controls crawling, not indexing. It is not a way to hide pages.
Robots.txt basics in two minutes
Each file is made of groups. A group starts with one or more User-agent lines and is followed by Allow and Disallow rules.
User-agent: *applies to every crawler that has no more specific group.Disallow: /path/blocks URLs that start with that path.Allow: /path/filere-opens a path inside a blocked folder.*matches any sequence of characters and$marks the end of a URL (both are supported by Google and Bing).Sitemap:gives the full URL of your XML sitemap and can appear anywhere in the file.- Lines starting with
#are comments.
Paths are case-sensitive. /Admin/ and /admin/ are different rules.
Allow all and disallow all
Allow all crawlers
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
An empty Disallow: means nothing is blocked. Having no robots.txt at all has the same effect, but a file lets you list your sitemap.
Disallow all crawlers
User-agent: *
Disallow: /
This blocks every compliant crawler from every URL. Use it only on sites that should never be crawled, and remember that blocked URLs can still appear in search results if other sites link to them.
Examples for common sites
Block one folder
User-agent: *
Disallow: /internal-reports/
Sitemap: https://example.com/sitemap.xml
The trailing slash blocks the folder and everything inside it. Without it, Disallow: /internal-reports would also block /internal-reports-2026.html.
WordPress
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xml
This matches the default that WordPress generates, plus a sitemap line. Do not block /wp-content/ or /wp-includes/: Google needs your CSS and JavaScript to render pages. If you use an SEO plugin, replace the sitemap URL with the one it creates (often /sitemap_index.xml).
Ecommerce filter and sort parameters
User-agent: *
Disallow: /*?*sort=
Disallow: /*?*color=
Disallow: /*?*price=
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Sitemap: https://shop.example.com/sitemap.xml
Filtered and sorted listing pages can create thousands of near-identical URLs. Blocking the parameters saves crawl activity for your real category and product pages. Before you add this, check that your canonical tags point filtered URLs to the main category page; the Canonical and Index Checker shows the canonical and robots meta for any URL. If a filtered page is one you want ranked (for example "red sarees under ₹2,000"), give it a clean static URL instead of blocking it.
Sitemap lines
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-products.xml
Sitemap: https://blog.example.com/sitemap.xml
Use absolute URLs. You can list several sitemaps. Not sure where yours is? The Sitemap Finder and Validator locates and checks it.
Staging or development site
# staging.example.com/robots.txt
User-agent: *
Disallow: /
This keeps compliant crawlers away from a test copy, but it is not protection. Anyone with the link can still open the site, and linked URLs can still be indexed. Put staging behind a password (HTTP authentication) and keep this file as a second layer. When you launch, do not copy the staging robots.txt to production. This mistake blocks entire live sites.
AI crawlers: allow or block
AI companies publish separate user-agent names, so you can make different choices for training, AI search and normal search.
| User agent | Operator | What it controls |
|---|---|---|
| GPTBot | OpenAI | Crawling for model training |
| OAI-SearchBot | OpenAI | Pages shown in ChatGPT search results |
| ChatGPT-User | OpenAI | Fetches triggered by a user's request in ChatGPT |
| Google-Extended | Product token for Gemini and Vertex AI training use; not a separate crawler, no effect on Google Search | |
| PerplexityBot | Perplexity | Crawling for Perplexity |
| ClaudeBot | Anthropic | Crawling by Anthropic |
| CCBot | Common Crawl | The open Common Crawl dataset, used by many AI projects |
| Applebot-Extended | Apple | Token controlling use of Apple-crawled content for Apple's AI training |
Block AI training, keep AI search
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
Block all listed AI crawlers
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Google-Extended
User-agent: PerplexityBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Disallow:
Several User-agent lines can share one group. Googlebot and Bingbot are not affected, so normal search continues.
Allow all AI crawlers
If you want maximum visibility in AI answers, the plain allow-all file above already permits them. You can still name them explicitly for clarity.
Two cautions. First, robots.txt is a request. Well-behaved crawlers follow it; a bad actor can ignore it, so use server or firewall rules for real blocking. Second, user-agent names change. Check each operator's documentation (OpenAI publishes its list on its crawler page) before you rely on a rule.
What robots.txt cannot do
It does not hide pages from search
Use noindex for that. Add <meta name="robots" content="noindex"> to the page, or send an X-Robots-Tag: noindex HTTP header for PDFs and other files. The page must stay crawlable, because Google has to fetch it to see the noindex. If you block it in robots.txt, Google never sees the tag.
It does not protect private data
The file is public. Anyone can read /robots.txt, so listing secret folders in it advertises them. Use login protection.
It does not work from a subfolder
The file must be at the root of the host, for example https://example.com/robots.txt. A file at https://example.com/blog/robots.txt is ignored. Each subdomain (shop.example.com, blog.example.com) needs its own file, and so does each protocol and port.
A robots.txt check is one item in our SEO audit checklist for 2026, which also covers canonicals, noindex tags and sitemaps.
How to test your robots.txt
- Build or edit the file. The Robots.txt Generator creates a clean file with your sitemap and crawl rules.
- Upload it to the root of your host as plain text (UTF-8), named exactly
robots.txtin lower case. - Open
https://yourdomain.com/robots.txtin a browser and confirm it loads with a 200 status. - Run the live file through the Robots.txt Tester to fetch and validate it.
- In Google Search Console, open the robots.txt report (under Settings) to see the version Google last fetched and any parse errors.
- Use the URL Inspection tool on two or three important pages to confirm they are not blocked.
Test after every CMS update, theme change or migration. These are the moments when a wrong file reaches production.
Next steps
- Pick the example above that matches your site and adjust the paths.
- Add a
Sitemap:line with the absolute URL. - Decide your AI crawler policy for training and for AI search separately.
- Use
noindex, not robots.txt, for pages you want out of search results. - Test the live file and recheck it after each deployment.
