How to Disallow All Except for robots.txt in Your Site Crawling Rules? or Is There a Way to Disallow

18 Replies, 1907 Views

Your code works for disallow all except robots.txt, but bad bots will bypass it.

Try combining it with IP blocking or CAPTCHAs for sketchy traffic. Tools like Distil or Imperva can help too.

Messages In This Thread



Users browsing this thread: 1 Guest(s)