+- Proxy Community (https://proxycommunity.com/forum)
+-- Forum: Use Case (https://proxycommunity.com/forum/forum-use-case)
+--- Forum: Web Scraping (https://proxycommunity.com/forum/forum-web-scraping)
+--- Thread: How to scrape a whole website's robots.txt file properly? or Best way to scrape a whole website's rob (/thread-how-to-scrape-a-whole-website-s-robots-txt-file-properly-or-best-way-to-scrape-a-whole-website-s-rob)