How to Disallow All Except for robots.txt in Your Site Crawling Rules? or Is There a Way to Disallow

18 Replies, 2131 Views

disallow all except for robots.txt is easy with that code, but it’s not foolproof.

Some bots just don’t care. You could also use server-level blocking or Cloudflare’s bot fight mode.

Also, check your logs to see who’s still crawling—might need to block IPs manually.

Messages In This Thread



Users browsing this thread: 1 Guest(s)