Scrapy is overkill unless you’re scraping a ton of sites. For one-off jobs, just use Python’s `urllib.request`.
Example:
```python
import urllib.request
url = "https://example.com/robots.txt"
response = urllib.request.urlopen(url)
print(response.read())
```
Simple, right? No need to complicate it.
Example:
```python
import urllib.request
url = "https://example.com/robots.txt"
response = urllib.request.urlopen(url)
print(response.read())
```
Simple, right? No need to complicate it.
