How to scrape a whole website's robots.txt file properly? or Best way to scrape a whole website's rob

For a quick fix, just wget it:

`wget --recursive --accept=txt --no-parent https://example.com/robots.txt`

Not fancy but gets the job done.

Rate limits? Add `--wait=5` to space out requests.