How to scrape all pages from a website using robot.txt – is it possible? or Can you scrape all pages

Title: Can you scrape all pages from a website by parsing robot.txt? How?

Hey folks!

So I’ve been digging into this whole *scrape all pages from website robot.txt how to* thing, and I’m kinda stuck. Like, I know robot.txt tells crawlers what’s off-limits, but can you actually use it to *find* all the pages?

Some say it’s possible if the sitemap’s listed there, but others say nah, it’s just for blocking. Anyone got experience with this?

Also, if it *is* doable, what’s the best way to go about it? Python? Some random tool? Or am I just wasting my time lol.

Thanks in advance!