Wow, thanks for all the insights, folks! Didn’t expect so many detailed responses.
I think I’ll give back-end scraping a shot with Scrapy + proxies, but I’ll keep it in a separate microservice like some of you suggested. The legal stuff is definitely spooking me a bit, so I’ll double-check robots.txt and add delays.
Quick follow-up: Anyone here had issues with Cloudflare while scraping? How’d you get around it?
Cheers! 🍻
I think I’ll give back-end scraping a shot with Scrapy + proxies, but I’ll keep it in a separate microservice like some of you suggested. The legal stuff is definitely spooking me a bit, so I’ll double-check robots.txt and add delays.
Quick follow-up: Anyone here had issues with Cloudflare while scraping? How’d you get around it?
Cheers! 🍻
