Wow, thanks for all the awesome replies, everyone! I’ve been using proxy with Scrapy for a while, but I’ve already picked up a few new tricks from this thread.
I’m definitely gonna check out some of the tools you guys mentioned, like ProxyMesh and Scrapy-Splash. I’ve been using Bright Data for proxies, but I’m curious about trying out residential proxies now.
One follow-up question: has anyone tried using machine learning to detect when a site is about to block you? I’ve heard some people are experimenting with it, but I’m not sure if it’s worth the effort.
Thanks again for all the tips—this community rocks!
I’m definitely gonna check out some of the tools you guys mentioned, like ProxyMesh and Scrapy-Splash. I’ve been using Bright Data for proxies, but I’m curious about trying out residential proxies now.
One follow-up question: has anyone tried using machine learning to detect when a site is about to block you? I’ve heard some people are experimenting with it, but I’m not sure if it’s worth the effort.
Thanks again for all the tips—this community rocks!