Honestly, should you do web scraping in the back end? Only if you’re prepared for the upkeep.
I’ve had scrapers running for months, then bam—site redesign, and everything breaks. It’s a time sink.
If you must, use a queue system (like Celery) to manage tasks. And cache aggressively to avoid repeat scrapes.
I’ve had scrapers running for months, then bam—site redesign, and everything breaks. It’s a time sink.
If you must, use a queue system (like Celery) to manage tasks. And cache aggressively to avoid repeat scrapes.