Hey, great thread! Using proxy with Scrapy is def a must, but it’s not just about the proxies. I’ve found that using a tool like Scrapy-Splash can help you handle JavaScript-heavy sites without getting blocked.
For proxies, I’ve had good luck with Luminati. They’re a bit pricey, but their IPs are super reliable, and they’ve got a ton of locations to choose from.
Oh, and don’t forget to use a custom middleware to handle cookies. Sometimes sites will block you if they detect that you’re not maintaining a session properly.
For proxies, I’ve had good luck with Luminati. They’re a bit pricey, but their IPs are super reliable, and they’ve got a ton of locations to choose from.
Oh, and don’t forget to use a custom middleware to handle cookies. Sometimes sites will block you if they detect that you’re not maintaining a session properly.