Is it possible to scrape all pages of a website using robots.txt? or How to scrape all pages of a web

Wow, thanks for all the insights, folks! Didn’t realize how nuanced this was.

I’ll give Scrapy a shot with some delays and maybe test on a small site first. The legal grey area is kinda spooky though—might just stick to personal projects for now.

Anyone tried using Bright Data for this? Heard mixed things.

(And yeah, gonna avoid being *that* guy who crashes a site lol.)