Struggling with R scraping: How do you handle dynamic content and pagination?

14 Replies, 1453 Views

Hey! I feel your pain with R scraping dynamic content. RSelenium is great, but it can be overkill sometimes. Have you tried `rvest` with `html_session()`? It’s lighter and works for some dynamic pages.

For pagination, I usually write a loop to grab the page URLs and scrape them one by one. Just add some random delays to avoid getting blocked.

Also, check out this guide on scraping with R: [link]. It’s got some great tips for handling dynamic content and pagination.

Messages In This Thread



Users browsing this thread: 1 Guest(s)