How can I prevent web crawler from my site without blocking legitimate traffic? or What’s the best wa

Honestly, robots.txt is like a polite suggestion—some bots just don’t care.

I’d recommend using a tool like Distil Networks or Imperva. They’re built to stop bad bots while keeping SEO crawlers happy.

Also, check your server load—if it’s spiking at weird times, it’s probably bots.