Wow, thanks for all the suggestions, everyone! I’ve been trying out Selenium with Bright Data proxies, and it’s been working pretty well so far. Still running into some captchas, but I’ll definitely check out FlareSolverr and Zenscrape for that.
Quick question though—has anyone tried Scrapy Cloud? I’ve heard mixed reviews, but it seems like it could handle some of the scaling issues I’m running into.
Also, thanks for the heads-up on the legal stuff. I’ve been double-checking robots.txt files like crazy now. 😅
Cheers for all the help!
Quick question though—has anyone tried Scrapy Cloud? I’ve heard mixed reviews, but it seems like it could handle some of the scaling issues I’m running into.
Also, thanks for the heads-up on the legal stuff. I’ve been double-checking robots.txt files like crazy now. 😅
Cheers for all the help!