Dynamic content is such a pain, but I’ve found that using ActionChains in Selenium helps. Like, hovering over elements or scrolling to trigger lazy-loaded content.
For AJAX, I’ll often wait for a specific element to change, like a counter or status message. If all else fails, I’ll use JavaScript to force-load stuff, but it’s not always reliable.
Have you tried Splash? It’s a headless browser that’s great for rendering JavaScript-heavy sites. Works well with Scrapy too!
For AJAX, I’ll often wait for a specific element to change, like a counter or status message. If all else fails, I’ll use JavaScript to force-load stuff, but it’s not always reliable.
Have you tried Splash? It’s a headless browser that’s great for rendering JavaScript-heavy sites. Works well with Scrapy too!