Wow, thanks for all the tips! I tried Selenium + proxies like some of you suggested, and it’s working way better. Still getting some timeouts tho—any idea how to tweak the delay settings?
Also, anyone used Scrapy Cloud? Wondering if it’s worth the $$$ for my project.
PS: The `requests-html` suggestion was gold—saved me hours!
Also, anyone used Scrapy Cloud? Wondering if it’s worth the $$$ for my project.
PS: The `requests-html` suggestion was gold—saved me hours!