Hey everyone! đź‘‹
So, I’ve been trying to scrape tweets for a project, and man, it’s been a rollercoaster. Twitter’s API changes in 2023 have made it trickier, but I’ve found a few tools that still work pretty well.
First off, Snscrape is my go-to for scraping tweets without hitting API limits. It’s lightweight and doesn’t require API keys, which is a lifesaver.
If you’re looking for something more GUI-based, Octoparse is decent, though it can be a bit slow.
For those who code, Tweepy with Twitter’s API v2 is solid, but you gotta deal with rate limits.
Oh, and don’t forget about Twint—it’s a bit hit or miss now, but when it works, it’s gold for scraping tweets.
Anyone else got tips or tools they’re using to scrape tweets in 2023? Would love to hear what’s working for y’all!
Cheers! 🍻
Yo, great thread! I’ve been using Scrapy with some custom middleware to scrape tweets, and it’s been working pretty well for me. It’s a bit of a learning curve, but once you get the hang of it, it’s super powerful.
Also, check out PhantomBuster if you’re into no-code solutions. It’s not free, but it’s reliable for scraping tweets and other social media data.
Anyone tried Apify? Heard it’s good for Twitter scraping too, but haven’t tested it yet.
Hey! Just wanted to add that TweetScraper is another Python library that’s been working for me. It’s not as popular as Tweepy, but it’s lightweight and gets the job done.
Also, if you’re okay with paid tools, Brand24 is pretty solid for scraping tweets and monitoring mentions. It’s more of a social listening tool, but it’s worth checking out.
Snscrape is def a lifesaver! I’ve been using it for a while now, and it’s been super reliable for scraping tweets without hitting limits.
For those who don’t wanna code, Social Searcher is a decent alternative. It’s web-based and lets you scrape tweets by keywords or hashtags. Not as flexible as coding your own scraper, but it’s quick and easy.
Twint is still my go-to for scraping tweets, even though it’s a bit finicky. When it works, it’s amazing. Just make sure you’re using the latest version and have all the dependencies installed.
Also, Scrapy Cloud is worth looking into if you want to scale your scraping projects. It’s not free, but it’s great for handling larger datasets.
Thanks for all the suggestions, everyone! I’ve been testing out Snscrape and Tweepy, and they’re both working great so far.
Quick question though—has anyone tried combining tools? Like using Snscrape for the initial scrape and then cleaning the data with Pandas? Curious if that’s overkill or if it’s worth the effort.
Also, shoutout to whoever mentioned DataMiner—gonna give that a shot for smaller projects. Cheers! 🍻
If you’re into R, rtweet is a solid package for scraping tweets. It works with Twitter’s API v2 and handles rate limits pretty well.
For a no-code option, Import.io is another tool I’ve used. It’s not specifically for Twitter, but you can set it up to scrape tweets and export the data easily.
Octoparse is cool, but yeah, it can be slow. I’ve been using ParseHub lately, and it’s been faster for scraping tweets. It’s also no-code, so it’s great for beginners.
Also, don’t sleep on DataMiner—it’s a browser extension that lets you scrape tweets directly from Twitter’s website. Super handy for quick projects.
Tweepy is solid, but the rate limits can be a pain. I’ve been using Twitter Scraper (a Python library) alongside it, and it’s been a game-changer.
If you’re looking for a cloud-based solution, ScrapingBee is worth checking out. It handles proxies and CAPTCHAs, which makes scraping tweets way easier.
For those who want a mix of coding and no-code, Zenscrape is a good option. It’s an API that lets you scrape tweets and other web data without worrying about IP bans.
Also, Mozenda is another tool I’ve heard good things about, though I haven’t tried it myself.
