How to Disallow All Except for robots.txt in Your Site Crawling Rules? or Is There a Way to Disallow

Your robots.txt example is correct for disallow all except for robots.txt! That’s the standard way to do it.

But yeah, some crawlers ignore it (looking at you, sketchy bots).

If you wanna go nuclear, block IPs or use .htaccess/server-side rules. Tools like Cloudflare or Bot Manager can help too.