Honestly, robot.txt isn’t gonna help you scrape all pages from a website. It’s more like a stop sign for crawlers. If you’re looking for tools, I’d recommend ParseHub or even Python with Selenium.
But yeah, be careful with the ethics. If the site owner doesn’t want you scraping, it’s kinda shady to go ahead anyway. Maybe try reaching out to them first?
But yeah, be careful with the ethics. If the site owner doesn’t want you scraping, it’s kinda shady to go ahead anyway. Maybe try reaching out to them first?