Robots.txt Generator
The robots.txt Generator creates robots.txt files for controlling search engine crawler access. The robots.txt file tells search engine crawlers which pages they can and cannot crawl. This generator lets you allow or disallow specific directories, set crawl delays, and reference your sitemap. Proper robots.txt configuration ensures search engines crawl the right pages and avoids wasting crawl budget on unimportant pages.
Formula
robots.txt format: User-agent: * Disallow: /private/ Allow: /public/ Sitemap: https://example.com/sitemap.xml Crawl-delay: 10 User-agent: Googlebot Disallow: /no-google/ Standard: robots.txt protocol
Example
Allow all crawling: User-agent: * Disallow: Block specific directory: User-agent: * Disallow: /admin/ Disallow: /private/ Block all crawling: User-agent: * Disallow: /
How to Use
- Choose which user agents to target (all or specific)
- Set disallow rules for directories to block
- Optionally set crawl delay and sitemap URL
- The generator creates the robots.txt content
- Upload to your website root directory
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file is a text file placed in your website root that tells search engine crawlers which pages they can and cannot access. It is the first file crawlers look for when visiting your site.
Where should I place robots.txt?
Place the robots.txt file in the root directory of your website. It must be accessible at https://yourdomain.com/robots.txt. For example, https://calczy.com/robots.txt.
Does robots.txt prevent indexing?
No. robots.txt prevents crawling but not indexing. Google may still index a page if it is linked from other sites, even if robots.txt blocks it. To prevent indexing, use the noindex meta tag or X-Robots-Tag HTTP header.
What is the Crawl-delay directive?
Crawl-delay sets the number of seconds between requests from a crawler. For example, Crawl-delay: 10 means the crawler waits 10 seconds between requests. Google ignores this directive, but Bing and Yandex respect it.
Should I block crawlers from certain pages?
Block crawlers from private, duplicate, or low-value pages like admin panels, search results pages, and staging environments. This conserves crawl budget for important pages. Do not block CSS, JS, or image files that search engines need to render your pages.