You’re kinda mixing things up—robots.txt tells crawlers what *not* to scrape, not how to find everything.
If you wanna scrape all pages from a website, try this:
1. Start at the homepage
2. Extract all links
3. Repeat for each new link
Scrapy’s great for this, or you can use Apify if you want a no-code option. Just don’t be a jerk with the requests, okay?
If you wanna scrape all pages from a website, try this:
1. Start at the homepage
2. Extract all links
3. Repeat for each new link
Scrapy’s great for this, or you can use Apify if you want a no-code option. Just don’t be a jerk with the requests, okay?