Wow, thanks for all the suggestions, everyone! I’ve been trying out Octoparse and it’s been working pretty well so far. Still getting the hang of it, but it’s def better than what I was using before.
I’m also curious about the Twitter API v2—has anyone here used it for large-scale projects? I’m a bit worried about the rate limits, but it sounds like it might be more reliable than scraping.
Also, has anyone tried using proxies with their scraping tools? I’ve heard mixed things about whether they help with avoiding bans.
Thanks again for all the tips—you guys are awesome! 🍵
I’m also curious about the Twitter API v2—has anyone here used it for large-scale projects? I’m a bit worried about the rate limits, but it sounds like it might be more reliable than scraping.
Also, has anyone tried using proxies with their scraping tools? I’ve heard mixed things about whether they help with avoiding bans.
Thanks again for all the tips—you guys are awesome! 🍵