Best way to scrape whole website robots.txt – how to do it properly? or How to scrape whole website r

18 Replies, 1400 Views

Pro tip: Check if the site even has a robots.txt before trying to scrape whole website robots.txt how to. Some don’t!

Use `requests.head()` first to see if it exists. Saves time and avoids unnecessary requests.

For parsing, `robotparser` (built into Python) is clutch. It’s literally made for this.

Messages In This Thread



Users browsing this thread: 1 Guest(s)