How to scrape a whole website's robots.txt file properly? or Best way to scrape a whole website's rob

lol i got banned once doing this.

Now I use Screaming Frog’s SEO Spider—it’s not free but handles robot.txt scraping whole website like a champ.

Also, their docs explain how to avoid blocks. Worth checking out if you’re doing this a lot.