Honestly, scraping all pages from a website using robots.txt is backwards—it’s a blocklist, not a checklist.
Try this instead:
- Crawl the sitemap if it exists
- Use a headless browser (like Selenium) to click through navigation
- Or just scrape Google’s cache of the site
But be cool—don’t overload the server, or you’ll ruin it for everyone.
Try this instead:
- Crawl the sitemap if it exists
- Use a headless browser (like Selenium) to click through navigation
- Or just scrape Google’s cache of the site
But be cool—don’t overload the server, or you’ll ruin it for everyone.