Wow, thanks for all the suggestions, everyone! I’ve been trying out Snscrape based on your recommendations, and it’s been working pretty well so far. Still figuring out the proxy situation—I’ll probably give Bright Data or Oxylabs a shot since they seem reliable.
Quick question though: for those using Scrapy, how do you handle CAPTCHAs? I’ve heard they can be a pain when scraping twitter at scale. Also, anyone tried using Twint recently? I’m curious if it’s still viable in 2023.
Thanks again for all the tips—this has been super helpful! 🍻
Quick question though: for those using Scrapy, how do you handle CAPTCHAs? I’ve heard they can be a pain when scraping twitter at scale. Also, anyone tried using Twint recently? I’m curious if it’s still viable in 2023.
Thanks again for all the tips—this has been super helpful! 🍻