Yup, 429s are the worst.
I’ve had success with `got-scraping` (for JS)—it handles retries and backoff automagically.
Another trick: delay aggressive scraping during peak hours. Servers are less forgiving when they’re busy.
Oh, and log your fails! Sometimes the pattern isn’t obvious until you see it all laid out.
I’ve had success with `got-scraping` (for JS)—it handles retries and backoff automagically.
Another trick: delay aggressive scraping during peak hours. Servers are less forgiving when they’re busy.
Oh, and log your fails! Sometimes the pattern isn’t obvious until you see it all laid out.
