Should You Do Web Scraping in the Back End? Pros, Cons, and Best Practices or Is It a Good Idea to Do

27 Replies, 2333 Views

Wow, thanks for all the insights, folks! Didn’t expect so many detailed responses.

I think I’ll give back-end scraping a shot with Scrapy + proxies, but I’ll keep it in a separate microservice like some of you suggested. The legal stuff is definitely spooking me a bit, so I’ll double-check robots.txt and add delays.

Quick follow-up: Anyone here had issues with Cloudflare while scraping? How’d you get around it?

Cheers! 🍻

Messages In This Thread



Users browsing this thread: 2 Guest(s)