How can I prevent web crawler from my site without blocking legitimate traffic? or What’s the best wa

Kinda surprised no one’s mentioned honeypots yet.

You can add invisible links or traps that only bots would follow. If something clicks ’em, boom—you’ve got a bot.

It’s a sneaky way to prevent web crawler from your site without affecting real users or SEO.

Just don’t overdo it, or you might catch something you didn’t mean to.