Is it possible to scrape all pages of a website using robots.txt? or How to scrape all pages of a web

Hey folks!

Quick question—anyone know if it's actually possible to *scrape all pages website robots.txt* allows it? Like, if the file doesn’t block certain paths, does that mean it’s fair game?

I’ve seen mixed opinions. Some say robots.txt is just a suggestion, others treat it like law.

Also, what’s the best way to *scrape all pages website robots.txt* without being a jerk about it? Slow crawling? Respecting crawl-delay?

And uh… legally speaking, if robots.txt gives the green light, are we in the clear? Or are there still hidden gotchas?

Thanks in advance! 🙏

*(ps. sorry for typos, typing this on my phone lol)*