Best way to scrape whole website robots.txt – how to do it properly? or How to scrape whole website r

18 Replies, 1401 Views

For a quick and dirty way to scrape whole website robots.txt how to, just use wget or curl in terminal:

```bash
wget example.com/robots.txt
```

No fancy tools needed if you’re just grabbing one file. But if you’re scaling up, yeah, Scrapy + proxies is the move.

Messages In This Thread



Users browsing this thread: 1 Guest(s)