Hey! Welcome to the wild world of scraping. 😄
First off, yeah, slowing down your requests is key. If you hammer a site too fast, they’ll block you in seconds. Try adding delays between requests—like 2-5 seconds.
For proxies, *absolutely* use them if you’re scraping a site at scale. Rotating IPs helps avoid bans. Check out Bright Data or Oxylabs—they’re pricey but solid.
Legal stuff? Grey area, tbh. If it’s personal use and you’re not violating their ToS, you’re *probably* fine. But always check robots.txt first!
Tool rec: Scrapy + rotating proxies is my go-to.
First off, yeah, slowing down your requests is key. If you hammer a site too fast, they’ll block you in seconds. Try adding delays between requests—like 2-5 seconds.
For proxies, *absolutely* use them if you’re scraping a site at scale. Rotating IPs helps avoid bans. Check out Bright Data or Oxylabs—they’re pricey but solid.
Legal stuff? Grey area, tbh. If it’s personal use and you’re not violating their ToS, you’re *probably* fine. But always check robots.txt first!
Tool rec: Scrapy + rotating proxies is my go-to.