Hey! I feel you on the dynamic content struggle. rvest and httr are great for static stuff, but yeah, JS-heavy sites can be a pain.
I’ve been using RSelenium for those cases, and while it’s a bit heavy, it gets the job done. For smaller projects, I’ve found V8 (from the V8 package) super handy—it lets you run JS directly in R without the overhead of Selenium.
Also, check out polite for ethical scraping—it’s a nice wrapper around httr that helps with rate-limiting and respecting robots.txt.
Hope that helps!
I’ve been using RSelenium for those cases, and while it’s a bit heavy, it gets the job done. For smaller projects, I’ve found V8 (from the V8 package) super handy—it lets you run JS directly in R without the overhead of Selenium.
Also, check out polite for ethical scraping—it’s a nice wrapper around httr that helps with rate-limiting and respecting robots.txt.
Hope that helps!