Wow, thanks so much for all the suggestions, everyone! I didn’t expect so many helpful replies.
I tried using a proxy service like BrightData, and it seems to be working better now. Still getting the error: scrape url [failed] occasionally, but way less often.
I’ll definitely look into Scrapy and Playwright too—sounds like they might be more reliable long-term.
Quick question though: for those using Puppeteer or Selenium, do you notice a big performance hit? Just curious if it’s worth the switch.
Thanks again, you all rock!
I tried using a proxy service like BrightData, and it seems to be working better now. Still getting the error: scrape url [failed] occasionally, but way less often.
I’ll definitely look into Scrapy and Playwright too—sounds like they might be more reliable long-term.
Quick question though: for those using Puppeteer or Selenium, do you notice a big performance hit? Just curious if it’s worth the switch.
Thanks again, you all rock!