For dynamic sites, I’ve had good luck with Pyppeteer (Python port of Puppeteer). It’s async, so it’s fast, and it handles how to scrape way pages are setup through a website with lazy loading like a champ.
Downside? The docs are kinda sparse, but there are decent tutorials out there.
Also, if the site has an API, you might not even need to scrape—check the network tab first!
Downside? The docs are kinda sparse, but there are decent tutorials out there.
Also, if the site has an API, you might not even need to scrape—check the network tab first!