Robot.txt is useless for finding pages unless the sitemap’s listed. And even then, it’s hit or miss.
If you’re serious about scraping all pages from website robot.txt how to, you’re better off crawling the site properly. Use Scrapy or even just wget with `--recursive`.
Or, if the site’s small, Screaming Frog’s free version might do the trick.
If you’re serious about scraping all pages from website robot.txt how to, you’re better off crawling the site properly. Use Scrapy or even just wget with `--recursive`.
Or, if the site’s small, Screaming Frog’s free version might do the trick.