I’ve been using puppeteer with render for a client project, and man, the timeouts KILL me.
Pro tip: set `waitUntil: 'networkidle2'` and bump up the timeout to like 60s. Also, chunk your scrapes—don’t try to do 100 pages in one go.
For heavy scraping, maybe look into ScrapingBee? They handle the headless browser stuff for you.
Pro tip: set `waitUntil: 'networkidle2'` and bump up the timeout to like 60s. Also, chunk your scrapes—don’t try to do 100 pages in one go.
For heavy scraping, maybe look into ScrapingBee? They handle the headless browser stuff for you.
