Honestly, the legal stuff around web scraping vs web crawling is what tripped me up.
Scraping can get you blocked (or worse) if you’re not careful with rate limits. Crawling’s usually safer since it’s just indexing.
Check robots.txt folks!
Scraping can get you blocked (or worse) if you’re not careful with rate limits. Crawling’s usually safer since it’s just indexing.
Check robots.txt folks!
