How to Disallow All Except for robots.txt in Your Site Crawling Rules? or Is There a Way to Disallow

18 Replies, 2120 Views

Yep, that’s how you do it! But like others said, shady crawlers might ignore it.

If you’re on Apache, you could also use .htaccess to block everything but robots.txt:
```
<Files *>
Deny from all
</Files>
<Files robots.txt>
Allow from all
</Files>
```
Works like a charm.

Messages In This Thread



Users browsing this thread: 1 Guest(s)