For massive crawls, I split the work. Do 10k URLs at a time in screaming_frogmacos, then merge the data later.
Way more stable, and you won’t lose progress if it crashes.
Also, Cloudflare or aggressive firewalls can mess with it—try whitelisting the IP if you can.
Way more stable, and you won’t lose progress if it crashes.
Also, Cloudflare or aggressive firewalls can mess with it—try whitelisting the IP if you can.
