Yo, this is such a cool topic! I’ve been experimenting with web crawlers to get the utility rate API too, and honestly, it’s a rabbit hole.
One thing I’d add is to use Selenium if the site is heavy on JavaScript. It’s slower than Scrapy, but it works when other tools fail.
Also, for rotating IPs, I use ProxyMesh. It’s cheap and gets the job done.
Oh, and don’t forget to set a polite crawl rate. Some sites will block you if you hit them too hard, even if you’re using proxies.
One thing I’d add is to use Selenium if the site is heavy on JavaScript. It’s slower than Scrapy, but it works when other tools fail.
Also, for rotating IPs, I use ProxyMesh. It’s cheap and gets the job done.
Oh, and don’t forget to set a polite crawl rate. Some sites will block you if you hit them too hard, even if you’re using proxies.