Robots.txt
Checks if robots.txt exists and is properly configured
Checks if robots.txt exists and is properly configured
| Rule ID | crawl/robots-txt |
| Category | Crawlability |
| Scope | Site-wide |
| Severity | error |
| Weight | 8/10 |
Solution
robots.txt tells search engines which pages to crawl. Place it at the root of your domain (example.com/robots.txt). Include your sitemap URL. Avoid blocking important resources (CSS, JS, images) that search engines need to render pages. Never use ‘Disallow: /’ unless you want to block all crawling. Use Google Search Console to test your robots.txt.
Enable / Disable
Disable this rule
squirrel.toml
toml[rules]
disable = ["crawl/robots-txt"]Disable all Crawlability rules
squirrel.toml
toml[rules]
disable = ["crawl/*"]Enable only this rule
squirrel.toml
toml[rules]
enable = ["crawl/robots-txt"]
disable = ["*"]