Wow, didn’t expect so many replies! Thanks y’all—this clears up a lot.
I’ll def try ScraperAPI and add delays like suggested. One follow-up: how do you handle sites that *change* their robots.txt mid-scrape? Like, what if they suddenly block a path you’re already crawling?
(Also, lol @ the "don’t be a jerk" advice—noted.)
I’ll def try ScraperAPI and add delays like suggested. One follow-up: how do you handle sites that *change* their robots.txt mid-scrape? Like, what if they suddenly block a path you’re already crawling?
(Also, lol @ the "don’t be a jerk" advice—noted.)