How to scrape a whole website's robots.txt file properly? or Best way to scrape a whole website's rob

20 Replies, 1963 Views

If you’re on Windows, HTTrack is a no-brainer for scrape whole website robot.txt how to stuff.

GUI makes it easy, and you can throttle requests in settings.

Just don’t forget to exclude non-txt files unless you wanna download the whole site lol.

Messages In This Thread



Users browsing this thread: 1 Guest(s)