"Best Method to Disallow All Except for robots.txt – Any Suggestions?"
hey guys, quick question—how do i disallow all except for robots.txt for web crawlers?
i wanna block everything BUT that file. tried messing with the robots.txt rules but not sure if i’m doing it right.
something like:
```
User-agent: *
Disallow: /
Allow: /robots.txt
```
...but does this actually work? or is there a better way?
also, do some crawlers just ignore this anyway? kinda frustrating lol.
thanks in advance!
hey guys, quick question—how do i disallow all except for robots.txt for web crawlers?
i wanna block everything BUT that file. tried messing with the robots.txt rules but not sure if i’m doing it right.
something like:
```
User-agent: *
Disallow: /
Allow: /robots.txt
```
...but does this actually work? or is there a better way?
also, do some crawlers just ignore this anyway? kinda frustrating lol.
thanks in advance!
