Wow, thanks for all the tips! I’ve been trying the randomized delays + user agent rotation, and it’s *kinda* working. Still getting some 429 http errors, but way fewer.
Gonna test out ScraperAPI and see if it’s worth the hype.
Quick Q: Anyone know if free proxy lists are even safe? I’m sketched out by them.
Also, big shoutout to the async suggestion—totally forgot about that. Time to dust off my asyncio skills!
Gonna test out ScraperAPI and see if it’s worth the hype.
Quick Q: Anyone know if free proxy lists are even safe? I’m sketched out by them.
Also, big shoutout to the async suggestion—totally forgot about that. Time to dust off my asyncio skills!