Hey! I’ve been scraping semrush with selenium for a while now. For dynamic elements, I use `find_elements_by_xpath` with waits. It’s not perfect, but it works most of the time.
To avoid blocks, I recommend using residential proxies. I’ve tried a few, and Smartproxy has been decent.
Also, if you’re open to alternatives, Scrapy + Splash is a solid combo for dynamic content. Might be worth a look!
To avoid blocks, I recommend using residential proxies. I’ve tried a few, and Smartproxy has been decent.
Also, if you’re open to alternatives, Scrapy + Splash is a solid combo for dynamic content. Might be worth a look!