Wow, thanks for all the suggestions, everyone! I’ve been trying out Scrapy with some custom middleware like someone mentioned, and it’s been working pretty well so far.
I’m still hitting some rate limits though, so I might give Apify or ScrapingBee a shot next. Also, the tip about rotating user agents is gold—definitely gonna try that.
Quick question: anyone have experience with using cloud functions (like AWS Lambda) to scrape reddit page? Wondering if it’s worth the setup. Cheers!
I’m still hitting some rate limits though, so I might give Apify or ScrapingBee a shot next. Also, the tip about rotating user agents is gold—definitely gonna try that.
Quick question: anyone have experience with using cloud functions (like AWS Lambda) to scrape reddit page? Wondering if it’s worth the setup. Cheers!