disallow all except for robots.txt is easy with that code, but it’s not foolproof.
Some bots just don’t care. You could also use server-level blocking or Cloudflare’s bot fight mode.
Also, check your logs to see who’s still crawling—might need to block IPs manually.
Some bots just don’t care. You could also use server-level blocking or Cloudflare’s bot fight mode.
Also, check your logs to see who’s still crawling—might need to block IPs manually.
