How to scrape all pages from a website using robot.txt – is it possible? or Can you scrape all pages

Thanks for all the replies! So basically, robot.txt is a dead end unless there’s a sitemap. Got it.

I tried checking a few sites, and some do have sitemaps listed. Used Python to scrape it, and it worked!

Still kinda slow though—any tips on speeding it up? Maybe threading in Scrapy? Or is there a faster tool?

Also, anyone know if Google Search Console’s sitemap data is accessible for scraping? Just curious.

Thanks again!