How to scrape all pages from a website using robots.txt? Need guidance! or Best way to scrape all pag

18 Replies, 1919 Views

Title: scrape all pages website robots.txt how to - need help!

Hey guys,

So I’m trying to scrape all pages from a website using robots.txt, but I’m kinda stuck.

I know robots.txt shows allowed/disallowed paths, but how do I actually use it to find *all* pages? Like, does it list every URL or just rules?

Also, if the site has a sitemap in robots.txt, does that make it easier? Or am I overcomplicating this?

Any tips or tools to scrape all pages website robots.txt how to properly? Don’t wanna mess up or get blocked lol.

Thanks in advance!

(PS: If this has been asked before, my bad – point me to the right thread!)

Messages In This Thread
How to scrape all pages from a website using robots.txt? Need guidance! or Best way to scrape all pag - by - 03-09-2024, 08:41 AM



Users browsing this thread: 1 Guest(s)