Honestly, scrape whole website (robots.txt) feels sketchy even if it’s "allowed." Some devs just forget to update robots.txt, and you might accidentally DOS their site.
If you *must*, use rate limiting (like 1 req/sec).
Tools? Check out ScraperAPI—it handles proxies/throttling for you.
If you *must*, use rate limiting (like 1 req/sec).
Tools? Check out ScraperAPI—it handles proxies/throttling for you.