How to scrape all pages from a website using robots.txt – is it possible? or Can you scrape all pages

"Can you scrape all pages from a website using robots.txt? How?"

Hey guys, kinda new to web scraping and was wondering—can you actually scrape all pages from a website robots.txt how to do it properly?

I’ve seen some folks say robots.txt just tells crawlers what *not* to scrape, but is there a way to use it to find *all* the pages instead? Like, reverse-engineer it or something?

Or am I totally off base here? Would love to hear if anyone’s tried this before.

Also, if robots.txt isn’t the way, what’s the best method to scrape all pages from a site without getting blocked?

Thanks in advance! 🙌