Hey! I’ve been doing doi web scraping for a while now, and here’s my two cents:
- Respect robots.txt. It’s there for a reason, and ignoring it can get you in hot water.
- Use Scrapy or BeautifulSoup. Both are solid, but Scrapy is better for larger projects.
- For rate limits, try using a delay between requests. I usually go with 2-3 seconds.
Also, check out the website ParseHub. It’s a no-code tool that’s great for beginners.
- Respect robots.txt. It’s there for a reason, and ignoring it can get you in hot water.
- Use Scrapy or BeautifulSoup. Both are solid, but Scrapy is better for larger projects.
- For rate limits, try using a delay between requests. I usually go with 2-3 seconds.
Also, check out the website ParseHub. It’s a no-code tool that’s great for beginners.