Hey there! If you’re looking to scrape all pages from website robot.txt how to, I’d suggest starting with Screaming Frog SEO Spider. It’s a great tool for crawling websites and respects robot.txt by default.
As for ethics, it’s always good to ask for permission if you’re scraping a lot of data. Some sites are cool with it, others aren’t. Just be respectful and don’t overload their servers.
As for ethics, it’s always good to ask for permission if you’re scraping a lot of data. Some sites are cool with it, others aren’t. Just be respectful and don’t overload their servers.