How can I prevent web crawler from my site without blocking legitimate traffic? or What’s the best wa

Rate limiting is your friend here. Set up rules in your .htaccess or NGINX config to throttle requests from the same IP.

For example, if someone’s hitting your site more than, say, 10 times a second, it’s probably a bot.

And yeah, tools like Botify or Screaming Frog can help you analyze traffic and spot the bad guys.