If you’re on Windows, HTTrack is a no-brainer for scrape whole website robot.txt how to stuff.
GUI makes it easy, and you can throttle requests in settings.
Just don’t forget to exclude non-txt files unless you wanna download the whole site lol.
GUI makes it easy, and you can throttle requests in settings.
Just don’t forget to exclude non-txt files unless you wanna download the whole site lol.
