What's the best way to build a Python web crawler for scraping large sites? or How can I optimize my

For dynamic content, playwright is a solid middle ground between Selenium and pure requests. It’s faster than Selenium and still handles JS well.

Another hack: Check if the site has a mobile version—sometimes they’re less JS-heavy and easier to scrape.