Hey! I feel your pain with using proxy with scrapy. It’s such a headache when you get blocked.
I’d recommend trying out Scrapy-rotating-proxy. It’s a middleware that handles IP rotation for you, and it’s pretty solid.
For headers, I always use fake-useragent to randomize them. It helps a ton. And yeah, random delays are a must—I set mine between 1-3 seconds.
Free proxies? Nah, they’re too unreliable. I switched to paid ones like Bright Data, and it’s been a game-changer. Worth the investment if you’re scraping at scale.
Good luck!
I’d recommend trying out Scrapy-rotating-proxy. It’s a middleware that handles IP rotation for you, and it’s pretty solid.
For headers, I always use fake-useragent to randomize them. It helps a ton. And yeah, random delays are a must—I set mine between 1-3 seconds.
Free proxies? Nah, they’re too unreliable. I switched to paid ones like Bright Data, and it’s been a game-changer. Worth the investment if you’re scraping at scale.
Good luck!