Scrapy + Splash is my go-to combo for scrapping info at scale. Splash renders JS, so you don’t have to worry about missing data.
If you’re lazy like me, Apify’s got pre-built scrapers that work well, but it’s paid. Worth it if you’re doing heavy lifting.
Also, ProTip: Rotate user agents and use proxies unless you wanna get blocked fast.
If you’re lazy like me, Apify’s got pre-built scrapers that work well, but it’s paid. Worth it if you’re doing heavy lifting.
Also, ProTip: Rotate user agents and use proxies unless you wanna get blocked fast.