Ugh, I feel you. *Scraping failures unknown* are the bane of my existence.
I’ve been using Zyte (formerly Scrapinghub) for some of my projects. They handle a lot of the heavy lifting, so you don’t have to worry about proxies or headers.
Also, try adding more randomness to your requests. Sites are getting better at detecting patterns.
I’ve been using Zyte (formerly Scrapinghub) for some of my projects. They handle a lot of the heavy lifting, so you don’t have to worry about proxies or headers.
Also, try adding more randomness to your requests. Sites are getting better at detecting patterns.