If you’re asking "should you do web scraping in the back end," the answer is: it depends.
For small, one-off tasks? Maybe not worth the hassle. For large-scale, automated stuff? Definitely.
Just don’t forget:
- Rate limits are your friend.
- Log errors properly—you’ll thank yourself later.
- Use a DB to store scraped data so you’re not redoing work.
Tools: Postgres for storage, Scrapy for scraping.
For small, one-off tasks? Maybe not worth the hassle. For large-scale, automated stuff? Definitely.
Just don’t forget:
- Rate limits are your friend.
- Log errors properly—you’ll thank yourself later.
- Use a DB to store scraped data so you’re not redoing work.
Tools: Postgres for storage, Scrapy for scraping.