What’s up! For doi web scraping, I’d say the biggest thing is to be respectful. Don’t overload servers, and always follow robots.txt.
Tools-wise, I use a combo of BeautifulSoup and Selenium. Selenium is great for dynamic content, but it’s slower than other tools.
If you’re worried about getting blocked, try using a VPN or rotating proxies. I’ve had good luck with Oxylabs.
Ethics are key—just don’t be a jerk about it, and you’ll be fine.
Tools-wise, I use a combo of BeautifulSoup and Selenium. Selenium is great for dynamic content, but it’s slower than other tools.
If you’re worried about getting blocked, try using a VPN or rotating proxies. I’ve had good luck with Oxylabs.
Ethics are key—just don’t be a jerk about it, and you’ll be fine.