Struggling with R scraping: How do you handle dynamic content and pagination?

14 Replies, 1401 Views

Yo, R scraping dynamic content is a beast lol. RSelenium is solid, but it can be slow. Have you tried `rvest` with `html_session()`? It’s lighter and works for some dynamic pages.

For pagination, I use a combo of `httr` and `rvest`. Just grab the page URLs and loop through ‘em. Add some random delays to avoid getting blocked.

Also, check out this tutorial on scraping with R: [link]. It’s got some great tips for handling dynamic content.

Messages In This Thread



Users browsing this thread: 1 Guest(s)