URL: /rules/crawl/robots-txt

---
title: "Robots.txt"
description: "Checks if robots.txt exists and is properly configured"
---

Checks if robots.txt exists and is properly configured

| | |
|---|---|
| **Rule ID** | `crawl/robots-txt` |
| **Category** | [Crawlability](/rules/crawl) |
| **Scope** | Site-wide |
| **Severity** | error |
| **Weight** | 8/10 |

## Solution

robots.txt tells search engines which pages to crawl. Place it at the root of your domain (example.com/robots.txt). Include your sitemap URL. Avoid blocking important resources (CSS, JS, images) that search engines need to render pages. Never use 'Disallow: /' unless you want to block all crawling. Use Google Search Console to test your robots.txt.

## Enable / Disable

### Disable this rule

```toml squirrel.toml
[rules]
disable = ["crawl/robots-txt"]
```

### Disable all Crawlability rules

```toml squirrel.toml
[rules]
disable = ["crawl/*"]
```

### Enable only this rule

```toml squirrel.toml
[rules]
enable = ["crawl/robots-txt"]
disable = ["*"]
```