Yep, that’s how you do it! But like others said, shady crawlers might ignore it.
If you’re on Apache, you could also use .htaccess to block everything but robots.txt:
```
<Files *>
Deny from all
</Files>
<Files robots.txt>
Allow from all
</Files>
```
Works like a charm.
If you’re on Apache, you could also use .htaccess to block everything but robots.txt:
```
<Files *>
Deny from all
</Files>
<Files robots.txt>
Allow from all
</Files>
```
Works like a charm.
