How can I prevent web crawler from my site without blocking legitimate traffic? or What’s the best wa

Title: What’s the best way to prevent web crawler from your site while keeping SEO intact?

Hey everyone,

I’ve been noticing a ton of weird bots hitting my site lately, and it’s messing with performance. I wanna prevent web crawler from your site, but I don’t wanna block legit traffic or hurt my SEO.

Anyone got tips? I’ve heard about tweaking robots.txt or using rate limiting, but not sure if that’s enough.

Also, how do you even tell the difference between bad crawlers and search engines? Feels like a fine line.

Thanks in advance!

(PS: Sorry for any typos, typing this on my phone lol)