Hey! I’ve been scraping website R for a while, and I’ve found rvest to be the most reliable for static content. For dynamic stuff, I’ve been using chromote, and it’s been working great.
For anti-scraping, I’d recommend using Tor with rotating IPs. Also, make sure to randomize your request headers and add delays between requests.
If you’re dealing with super aggressive sites, maybe look into cloudflareR or Scrapy with reticulate.
For anti-scraping, I’d recommend using Tor with rotating IPs. Also, make sure to randomize your request headers and add delays between requests.
If you’re dealing with super aggressive sites, maybe look into cloudflareR or Scrapy with reticulate.