For big crawls, I tweak screaming_frogmacos like this:
- Threads: 6 (more than that and it chokes)
- Delay: 2s (avoids getting blocked)
- Exclude images/PDFs unless you NEED them.
JS is still hit or miss—try coupling it with Screaming Frog’s built-in rendering or just use Puppeteer for JS-heavy stuff.
- Threads: 6 (more than that and it chokes)
- Delay: 2s (avoids getting blocked)
- Exclude images/PDFs unless you NEED them.
JS is still hit or miss—try coupling it with Screaming Frog’s built-in rendering or just use Puppeteer for JS-heavy stuff.
