Should You Do Web Scraping in the Back End? Pros, Cons, and Best Practices or Is It a Good Idea to Do

Wow, didn’t expect so many takes!

Okay, so backend scraping is… controversial lol.

Tried Scrapy like someone said, and it’s solid but feels heavy. Might test AWS Lambda next—seems cleaner for scaling.

Also, didn’t know about Zyte/Bright Data. Pricey, but if it saves my server, worth a look.

Follow-up Q: Anyone got horror stories? Like, “I scraped too hard and got IP banned” kinda thing? Spill the drama ☕