Thanks everyone! Super helpful. I’ll try the .htaccess method since some bots are ignoring the robots.txt rules.
Quick follow-up: anyone know if Cloudflare’s bot protection is worth it for this? Or should I just stick to server-side blocking?
Also, my logs are a mess—any tools to help filter out legit vs. bad bots?
Quick follow-up: anyone know if Cloudflare’s bot protection is worth it for this? Or should I just stick to server-side blocking?
Also, my logs are a mess—any tools to help filter out legit vs. bad bots?
