How can I prevent web crawler from my site without blocking legitimate traffic? or What’s the best wa

Hey! Dealing with the same issue here. One thing that worked for me was using Cloudflare’s bot management. It’s pretty good at filtering out bad bots while letting Google and other legit crawlers through.

Also, tweaking robots.txt can help, but it’s not foolproof since some bots just ignore it. Maybe try rate limiting + Cloudflare?

For telling the difference, check your server logs—bad bots usually have weird user agents or hit the same pages over and over.