Hey, I’ve been down this rabbit hole before. For scrape all pages website robots.txt how to, robots.txt alone won’t cut it.
Sitemap is the way to go if it’s there. Try `curl` to fetch it or use a crawler like Scrapy.
If no sitemap, you’ll need to spider the site—but go slow or you’ll get IP banned.
Sitemap is the way to go if it’s there. Try `curl` to fetch it or use a crawler like Scrapy.
If no sitemap, you’ll need to spider the site—but go slow or you’ll get IP banned.