---
Hey y’all,
So, I’ve been thinking about trying to scrape IndieHackers for some data (don’t @ me, I’m just curious lol). Has anyone else tried to scrape IndieHackers before? Like, what tools did you use?
I’ve heard mixed things about BeautifulSoup and Scrapy, but idk if they’re overkill or not. Also, does IndieHackers even allow scraping? I don’t wanna get banned or anything.
If you’ve managed to scrape IndieHackers successfully, pls share your secrets. Or if it’s a bad idea, tell me why before I waste my weekend on this.
Thx in advance!
---
Hey! I’ve actually tried to scrape IndieHackers before, and it’s not as straightforward as it seems. I used BeautifulSoup and Requests in Python, but you gotta be careful with the rate limits. IndieHackers has some protections in place, so if you go too fast, you’ll get blocked.
I’d recommend starting slow and maybe using a proxy service like BrightData to avoid getting banned. Also, check their TOS—scraping might not be explicitly allowed, so tread lightly.
Good luck!
Lol, scraping IndieHackers sounds like a fun weekend project. I used Scrapy for a similar project, and it worked pretty well. It’s a bit more complex than BeautifulSoup, but it’s way more powerful if you’re dealing with a lot of pages.
Just make sure to set a delay between requests so you don’t get flagged. And yeah, double-check their terms—some sites are cool with scraping as long as you’re not hammering their servers.
I wouldn’t recommend scraping IndieHackers unless you’re super careful. They’ve got some pretty aggressive bot detection, and you could get IP banned real quick.
If you’re set on doing it, maybe try using Puppeteer or Playwright. They’re headless browsers that can mimic human behavior better than traditional scraping tools. Still, it’s risky, so maybe look for an API or dataset that’s already out there?
Scraping IndieHackers? Been there, done that. I used a combo of Selenium and BeautifulSoup. Selenium helps with the dynamic content, but it’s slower than other tools.
Honestly, it’s a bit of a gray area ethically and legally. If you’re just doing it for personal use and not spamming, you’re probably fine. But if you’re planning to monetize the data, you might wanna think twice.
Yo, I’ve tried to scrape IndieHackers before, and it’s a pain. Their site is pretty well-protected, and you’ll likely hit a wall if you don’t know what you’re doing.
I’d suggest using a tool like Octoparse—it’s no-code and easier to set up than coding your own scraper. But again, be mindful of their terms. Maybe reach out to their team and ask if it’s cool?
Scraping IndieHackers is doable, but it’s not for the faint of heart. I used Scrapy with rotating proxies to avoid getting blocked. It worked, but it took forever to set up.
If you’re just curious and not looking to scrape a ton of data, maybe try manually copying what you need? It’s tedious, but it’s safer than risking a ban.
I’ve heard mixed things about scraping IndieHackers too. Some people say it’s fine as long as you’re respectful, but others have gotten banned for it.
If you’re gonna try, maybe use a tool like ParseHub. It’s user-friendly and lets you scrape without writing code. Just keep the requests slow and steady.
Thanks for all the advice, y’all! I think I’m gonna give it a shot with BeautifulSoup and see how far I can get. I’ll definitely add some delays and maybe use a proxy to avoid getting banned.
Quick question though—has anyone tried using their API? I saw something about it but couldn’t find much info. If it’s legit, that might be a better route than scraping IndieHackers directly.
Also, if I run into any issues, I’ll update here. Appreciate the help!
Scraping IndieHackers is tricky, but not impossible. I used a combo of Requests and BeautifulSoup, and it worked okay. Just make sure to add random delays between requests to avoid detection.
Also, consider using a VPN or proxy to mask your IP. And yeah, check their TOS—scraping might not be allowed, so proceed at your own risk.
