Hey! Using proxy with scrapy is tricky, but once you get the hang of it, it’s smooth sailing.
I’d suggest using Scrapy-proxy-pool. It’s a great tool for rotating IPs without tripping alarms.
For headers, I use scrapy-user-agents. It’s super easy to set up and keeps things random.
Delays? I stick to 2-4 seconds. Too short, and you’ll get blocked; too long, and it’s inefficient.
As for proxies, I’d say go paid. Free ones are just too unreliable. I use Smartproxy, and it’s been solid.
Good luck with your project!
I’d suggest using Scrapy-proxy-pool. It’s a great tool for rotating IPs without tripping alarms.
For headers, I use scrapy-user-agents. It’s super easy to set up and keeps things random.
Delays? I stick to 2-4 seconds. Too short, and you’ll get blocked; too long, and it’s inefficient.
As for proxies, I’d say go paid. Free ones are just too unreliable. I use Smartproxy, and it’s been solid.
Good luck with your project!