Wow, thanks for all the replies! Didn’t realize robots.txt was the wrong approach—glad I asked before wasting time on it.
Gonna try the sitemap.xml trick first, and if that doesn’t work, I’ll test out Scrapy or Screaming Frog.
Quick follow-up: how do you handle sites with infinite scroll or lazy-loaded content? Do tools like Octoparse work for those, or do I need something fancier?
Appreciate the help! 🙏
Gonna try the sitemap.xml trick first, and if that doesn’t work, I’ll test out Scrapy or Screaming Frog.
Quick follow-up: how do you handle sites with infinite scroll or lazy-loaded content? Do tools like Octoparse work for those, or do I need something fancier?
Appreciate the help! 🙏