Best way to scrape whole website robots.txt – how to do it properly? or How to scrape whole website r

18 Replies, 1402 Views

Scrapy is overkill unless you’re scraping a ton of sites. For one-off jobs, just use Python’s `urllib.request`.

Example:
```python
import urllib.request
url = "https://example.com/robots.txt"
response = urllib.request.urlopen(url)
print(response.read())
```

Simple, right? No need to complicate it.

Messages In This Thread



Users browsing this thread: 1 Guest(s)