How to scrape a whole website's robots.txt file properly? or Best way to scrape a whole website's rob

20 Replies, 1954 Views

Hey! I’ve used BeautifulSoup + Requests for this before, but yeah, it’s slow for bulk scraping.

Try `scrape` (npm package) if you’re okay with Node.js—it’s way faster for multiple sites.

And for blocks? Rotate user-agents and use proxies. Free ones often suck though, so maybe invest in a cheap proxy service.

Messages In This Thread



Users browsing this thread: 1 Guest(s)