I’ve been using SiteCrawler for scraping dynamic content, and it’s decent but not perfect. For heavy JS sites, I’ve had better luck with Puppeteer or Playwright.
SiteCrawler does okay if you tweak the delay settings, but it’s not as reliable as a full browser-based crawler. If you’re dealing with SPAs, you might wanna check out Scrapy + Splash—it’s a bit more work to set up but handles dynamic stuff way better.
SiteCrawler does okay if you tweak the delay settings, but it’s not as reliable as a full browser-based crawler. If you’re dealing with SPAs, you might wanna check out Scrapy + Splash—it’s a bit more work to set up but handles dynamic stuff way better.