Wow, thanks for all the insights, everyone! I didn’t realize there were so many tools out there for scraping of information that also focus on ethics. I’ll definitely check out Scrapy and Octoparse.
I’m still a bit torn on the whole personal data thing, though. Like, how do you guys handle it when you’re scraping and accidentally come across something that feels too personal? Do you just stop and move on, or is there a way to filter that out beforehand?
Also, has anyone had any issues with sites blocking them for scraping? Just curious how common that is. Thanks again for all the advice!
I’m still a bit torn on the whole personal data thing, though. Like, how do you guys handle it when you’re scraping and accidentally come across something that feels too personal? Do you just stop and move on, or is there a way to filter that out beforehand?
Also, has anyone had any issues with sites blocking them for scraping? Just curious how common that is. Thanks again for all the advice!