Need Help Building a Node Website Scraper – Any Tips or Best Practices?

7 Replies, 1983 Views

Wow, thanks for all the tips, y’all! I tried Puppeteer with `puppeteer-extra-plugin-stealth`, and it’s working way better than I expected. Still struggling a bit with CAPTCHAs, but I’ll give 2Captcha a shot.

Quick question though—anyone know if there’s a way to detect if a site is blocking me without getting a full-on error? Like, maybe checking response headers or something?

Also, shoutout to whoever mentioned `bottleneck`—that’s been a game-changer for rate limits. Thanks again, everyone! ✌️

Messages In This Thread



Users browsing this thread: 1 Guest(s)