Scrapy + Splash is my go-to for how to scrape way pages are setup through a website with heavy JS. Splash renders the page, so you get the full HTML after all the scripts run.
It’s a bit of a setup hassle, but once it’s running, it’s smooth sailing.
Bonus: Scrapy’s built-in throttling and retries save you from IP bans.
It’s a bit of a setup hassle, but once it’s running, it’s smooth sailing.
Bonus: Scrapy’s built-in throttling and retries save you from IP bans.