How to scrape all pages from a website using robots.txt? Need guidance! or Best way to scrape all pag

18 Replies, 1931 Views

Yo, robots.txt ain’t gonna hand you all the URLs—it’s just rules. But if the site has a sitemap (often linked in robots.txt), you’re in luck!

For scrape all pages website robots.txt how to, I’d recommend:
- Parse the sitemap with Python’s `requests` + `xmltodict`
- Or use a tool like ParseHub if you’re not into coding

Just don’t hammer the server too hard, or you’ll get booted.

Messages In This Thread



Users browsing this thread: 1 Guest(s)