Yo, I feel you on the clunky scripts. I was in the same boat until I found Scrapy with rotating residential proxy pyproxy integration. It’s a beast for scraping and handles proxies way better.
For request frequency, I’d say start slow—like 1 request every 2-3 seconds—and then ramp up if the site doesn’t block you. Testing is key, so maybe use a site like httpbin.org to see if your IPs are rotating properly.
For request frequency, I’d say start slow—like 1 request every 2-3 seconds—and then ramp up if the site doesn’t block you. Testing is key, so maybe use a site like httpbin.org to see if your IPs are rotating properly.