Robots.txt Generator
Create a robots.txt file to control how search engines crawl your site
Generated Robots.txt
Save this as a file named "robots.txt" and upload it to the root directory of your website (e.g., https://example.com/robots.txt).
About Robots.txt
What is a Robots.txt File?
A robots.txt file is a text file webmasters create to instruct web robots (typically search engine crawlers) how to crawl pages on their website. The robots.txt file is part of the Robots Exclusion Protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.
Common Directives
| Directive | Description | Example |
|---|---|---|
| User-agent: | Specifies which search engine robot the rule applies to | User-agent: Googlebot |
| Disallow: | Tells the robot not to visit specified pages or directories | Disallow: /admin/ |
| Allow: | Tells the robot it can access a page or directory even if its parent directory is disallowed | Allow: /admin/public/ |
| Sitemap: | Specifies the location of the site's XML sitemap | Sitemap: https://example.com/sitemap.xml |
| Crawl-delay: | Specifies the number of seconds to wait between successive crawl requests | Crawl-delay: 10 |
Important Considerations
-
Not a security measure. Robots.txt is not a mechanism for keeping a web page out of search results. For that purpose, use the
noindexdirective or password protection. - Must be in the root directory. The robots.txt file must be located at the root of your site (e.g., https://example.com/robots.txt).
-
Case sensitivity. Directives in robots.txt are case-sensitive. The path
/private/is not the same as/Private/. - Pattern matching. Robots.txt supports limited pattern matching, such as using asterisks (*) as wildcards.
- Not all robots obey robots.txt. Some robots, especially malicious ones, may ignore your robots.txt file.
Frequently Asked Questions
No, robots.txt only prevents crawling, not indexing. Even if a page is blocked in robots.txt, it can still appear in search results if other pages link to it. To prevent a page from appearing in search results, use the
noindex meta tag or X-Robots-Tag HTTP header.
A robots.txt file is not strictly necessary, but it's considered a best practice, especially for larger websites. If you don't have a robots.txt file, search engines will crawl all publicly available pages of your site. You only need it if you want to restrict crawling of certain content.
If you block important pages with robots.txt, search engines won't crawl them, which means they won't be able to process the content on those pages. This can negatively impact your SEO if the pages contain valuable content. Make sure you only block pages that you genuinely don't want search engines to crawl, such as admin areas, private content, or duplicate content.
Use crawl-delay with caution. It instructs search engine bots to wait a specified number of seconds between requests to your server. This can be helpful if your server can't handle many requests at once, but it will slow down the crawling of your site, which could delay the indexing of new or updated content. Most modern websites don't need to use crawl-delay unless they have specific server limitations.
Google Search Console provides a robots.txt tester tool that allows you to check if your robots.txt file correctly allows or blocks specific URLs. Simply enter the URL you want to test, and the tool will tell you if it's allowed or disallowed according to your robots.txt rules. This is a good way to ensure your rules are working as intended before implementing them on your live site.