How to scrape all pages from a website using robots.txt? Need guidance! or Best way to scrape all pag

robots.txt is more like a "do not enter" sign, not a map of the whole site lol. For scrape all pages website robots.txt how to, you’re better off using the sitemap.xml if it’s there.

Try Scrapy or BeautifulSoup if you’re coding it yourself. Or just use Octoparse for a no-code option.

Pro tip: Set a delay between requests to avoid tripping alarms.