Lol, scraping IndieHackers sounds like a fun weekend project. I used Scrapy for a similar project, and it worked pretty well. It’s a bit more complex than BeautifulSoup, but it’s way more powerful if you’re dealing with a lot of pages.
Just make sure to set a delay between requests so you don’t get flagged. And yeah, double-check their terms—some sites are cool with scraping as long as you’re not hammering their servers.
Just make sure to set a delay between requests so you don’t get flagged. And yeah, double-check their terms—some sites are cool with scraping as long as you’re not hammering their servers.