Had the same website crawler python 403 forbidden drama. Proxies + headers + delays are the holy trinity.
But honestly, some sites just hate scrapers. If it’s critical, maybe look into their API? Or use a service like Apify to handle the headaches.
---
Glad I’m not the only one dealing with this! Tried the `fake-useragent` tip and it worked on a couple sites, but others still block me.
Gonna test out ScraperAPI next—thanks for the rec!
Anyone know if rotating IPs + selenium is overkill for a simple crawler? Or is that the only way now?
Cheers for all the help!
But honestly, some sites just hate scrapers. If it’s critical, maybe look into their API? Or use a service like Apify to handle the headaches.
---
Glad I’m not the only one dealing with this! Tried the `fake-useragent` tip and it worked on a couple sites, but others still block me.
Gonna test out ScraperAPI next—thanks for the rec!
Anyone know if rotating IPs + selenium is overkill for a simple crawler? Or is that the only way now?
Cheers for all the help!