Hey everyone,
So, I’ve been trying to figure out the *best way to scrape all pages from a website using robot.txt how to* do it properly. Like, I know robot.txt is supposed to guide crawlers, but can it actually help me scrape all pages from a website? Or is it just there to block stuff?
I’ve seen some tools that claim to do it, but idk if they’re legit or just gonna get me blocked lol. Anyone got tips on how to scrape all pages from a website robot.txt how to make it work without getting banned?
Also, is it even ethical? I mean, I’m not trying to spam or anything, just need the data for a project.
Pls share your thoughts or tools you’ve used! Thx in advance.
- A confused scraper
So, I’ve been trying to figure out the *best way to scrape all pages from a website using robot.txt how to* do it properly. Like, I know robot.txt is supposed to guide crawlers, but can it actually help me scrape all pages from a website? Or is it just there to block stuff?
I’ve seen some tools that claim to do it, but idk if they’re legit or just gonna get me blocked lol. Anyone got tips on how to scrape all pages from a website robot.txt how to make it work without getting banned?
Also, is it even ethical? I mean, I’m not trying to spam or anything, just need the data for a project.
Pls share your thoughts or tools you’ve used! Thx in advance.
- A confused scraper