Wow, thanks so much for all the replies, everyone! This is super helpful. I’ve been playing around with chromedp based on your suggestions, and it’s definitely handling the dynamic content better than what I was doing before.
I’m still struggling a bit with rate limits though—I tried setting delays in Colly, but I think I need to look into proxy services like ScrapingBee or Bright Data.
Quick question: does anyone have experience with Ferret? I saw it mentioned and it looks interesting, but I’m not sure if it’s worth the learning curve.
Thanks again, y’all are lifesavers! 🙌
I’m still struggling a bit with rate limits though—I tried setting delays in Colly, but I think I need to look into proxy services like ScrapingBee or Bright Data.
Quick question: does anyone have experience with Ferret? I saw it mentioned and it looks interesting, but I’m not sure if it’s worth the learning curve.
Thanks again, y’all are lifesavers! 🙌