Man, I feel you on the getting blocked part. Using proxy with Scrapy is a must, but it’s not foolproof. I’ve found that using a mix of free and paid proxies works best for me.
For free proxies, I scrape lists from sites like FreeProxyList or ProxyScrape, but I always test them with a tool like ProxyMule before using them in my scraper.
Also, don’t underestimate the power of session persistence. Sometimes sticking with the same IP for a bit (with proper delays) can actually help you avoid detection.
For free proxies, I scrape lists from sites like FreeProxyList or ProxyScrape, but I always test them with a tool like ProxyMule before using them in my scraper.
Also, don’t underestimate the power of session persistence. Sometimes sticking with the same IP for a bit (with proper delays) can actually help you avoid detection.