Thanks for all the replies, everyone! Super helpful. I tried Scrapy like a few of you suggested, and it’s working pretty well so far. I set up some delays to avoid getting blocked, and it’s scraping all pages from the website without any issues.
One quick follow-up: how do you guys handle sites that block you even with delays? Is there a way to rotate IPs or something? Or should I just move on to another site?
Also, big thanks for the ethical advice—I’ll make sure to keep it respectful and not overload their servers. Cheers!
One quick follow-up: how do you guys handle sites that block you even with delays? Is there a way to rotate IPs or something? Or should I just move on to another site?
Also, big thanks for the ethical advice—I’ll make sure to keep it respectful and not overload their servers. Cheers!