Wow, thanks so much for all the tips, y’all! I’ve been trying out BeautifulSoup + Requests for static content, and it’s working pretty well so far.
I did run into some issues with JS-heavy sites, though, so I’ll definitely give Selenium or Playwright a shot. Also, the proxy and user agent rotation tips are super helpful—I was getting blocked way too often.
Quick question: anyone have experience with Scrapy vs. Playwright for large-scale projects? Which one would you recommend for efficiency?
Thanks again, everyone! 🙌
I did run into some issues with JS-heavy sites, though, so I’ll definitely give Selenium or Playwright a shot. Also, the proxy and user agent rotation tips are super helpful—I was getting blocked way too often.
Quick question: anyone have experience with Scrapy vs. Playwright for large-scale projects? Which one would you recommend for efficiency?
Thanks again, everyone! 🙌