Title: scrape all pages website robots.txt how to - need help!
Hey guys,
So I’m trying to scrape all pages from a website using robots.txt, but I’m kinda stuck.
I know robots.txt shows allowed/disallowed paths, but how do I actually use it to find *all* pages? Like, does it list every URL or just rules?
Also, if the site has a sitemap in robots.txt, does that make it easier? Or am I overcomplicating this?
Any tips or tools to scrape all pages website robots.txt how to properly? Don’t wanna mess up or get blocked lol.
Thanks in advance!
(PS: If this has been asked before, my bad – point me to the right thread!)
Hey guys,
So I’m trying to scrape all pages from a website using robots.txt, but I’m kinda stuck.
I know robots.txt shows allowed/disallowed paths, but how do I actually use it to find *all* pages? Like, does it list every URL or just rules?
Also, if the site has a sitemap in robots.txt, does that make it easier? Or am I overcomplicating this?
Any tips or tools to scrape all pages website robots.txt how to properly? Don’t wanna mess up or get blocked lol.
Thanks in advance!
(PS: If this has been asked before, my bad – point me to the right thread!)
