Hey, robot.txt is more of a guideline for what you *shouldn’t* scrape, not a roadmap for scraping all pages from a website.
I’ve had good luck with DataMiner for smaller projects. It’s super user-friendly and doesn’t require coding. For bigger stuff, Scrapy is the way to go, but you’ll need to tweak the settings to avoid getting banned.
Ethics? If you’re not causing harm or stealing, it’s probably okay. Just don’t be a jerk about it.
I’ve had good luck with DataMiner for smaller projects. It’s super user-friendly and doesn’t require coding. For bigger stuff, Scrapy is the way to go, but you’ll need to tweak the settings to avoid getting banned.
Ethics? If you’re not causing harm or stealing, it’s probably okay. Just don’t be a jerk about it.