How to Disallow All Except for robots.txt in Your Site Crawling Rules? or Is There a Way to Disallow

18 Replies, 1833 Views

"Best Method to Disallow All Except for robots.txt – Any Suggestions?"

hey guys, quick question—how do i disallow all except for robots.txt for web crawlers?

i wanna block everything BUT that file. tried messing with the robots.txt rules but not sure if i’m doing it right.

something like:
```
User-agent: *
Disallow: /
Allow: /robots.txt
```
...but does this actually work? or is there a better way?

also, do some crawlers just ignore this anyway? kinda frustrating lol.

thanks in advance!

Messages In This Thread
How to Disallow All Except for robots.txt in Your Site Crawling Rules? or Is There a Way to Disallow - by - 25-10-2024, 07:54 PM



Users browsing this thread: 1 Guest(s)