Yo, scraping all pages from website robot.txt how to is a common question, but honestly, robot.txt isn’t the tool for scraping. It’s more like a guideline for bots. If you wanna scrape, you’ll need something like Octoparse or ParseHub. They’re super easy to use and don’t require coding.
Also, yeah, ethics matter. If the site says no scraping, don’t do it. You don’t wanna mess with someone’s server or get your IP banned.
Also, yeah, ethics matter. If the site says no scraping, don’t do it. You don’t wanna mess with someone’s server or get your IP banned.