How to Disallow All Except for robots.txt in Your Site Crawling Rules? or Is There a Way to Disallow

18 Replies, 2107 Views

Thanks everyone! Super helpful. I’ll try the .htaccess method since some bots are ignoring the robots.txt rules.

Quick follow-up: anyone know if Cloudflare’s bot protection is worth it for this? Or should I just stick to server-side blocking?

Also, my logs are a mess—any tools to help filter out legit vs. bad bots?

Messages In This Thread



Users browsing this thread: 1 Guest(s)