Hey! Been there—super frustrating when your web scrape script gets blocked even with delays.
One thing that worked for me: randomizing request headers *beyond* just user-agent. Try adding Accept-Language and Referer headers too. Also, check if the site has a robots.txt—sometimes they block certain paths.
For proxies, I’ve had decent luck with Luminati (now Bright Data), but they’re pricey. For free options, try ScraperAPI—it handles rotations for you.
Ever tried adding CAPTCHA solvers like 2Captcha? Some sites sneak those in.
One thing that worked for me: randomizing request headers *beyond* just user-agent. Try adding Accept-Language and Referer headers too. Also, check if the site has a robots.txt—sometimes they block certain paths.
For proxies, I’ve had decent luck with Luminati (now Bright Data), but they’re pricey. For free options, try ScraperAPI—it handles rotations for you.
Ever tried adding CAPTCHA solvers like 2Captcha? Some sites sneak those in.