How to scrape a whole website's robots.txt file properly? or Best way to scrape a whole website's rob

Honestly, just use Google’s cache if you only need a snapshot:

`site:example.com filetype:txt robots`

Not real-time but avoids blocks entirely.