Hey! Been using a javascript web scraper llm setup for a while now.
Puppeteer + LLM is my go-to, but yeah, it’s slow. For AJAX/React, you gotta wait for network idle.
Rate limits: I use a combo of delays and proxies. Also, don’t forget to handle CAPTCHAs—some sites throw them randomly.
Biggest pro? LLMs can handle nested, inconsistent data like no other. Worth the speed trade-off IMO.
Puppeteer + LLM is my go-to, but yeah, it’s slow. For AJAX/React, you gotta wait for network idle.
Rate limits: I use a combo of delays and proxies. Also, don’t forget to handle CAPTCHAs—some sites throw them randomly.
Biggest pro? LLMs can handle nested, inconsistent data like no other. Worth the speed trade-off IMO.