How to Disallow All Except for robots.txt in Your Site Crawling Rules? or Is There a Way to Disallow

18 Replies, 2103 Views

Yep, that’s the classic method for disallow all except robots.txt. But like everyone’s saying, it’s not 100%.

For tighter control, use .htaccess or server configs to physically block access to other files.

Also, check out Bot Protection services—they’re worth it if you’re getting hammered.

Messages In This Thread



Users browsing this thread: 1 Guest(s)