You got it! Crawling = discovery, scraping = extraction.
For a beginner, stick to scraping single sites. Tools:
- Browser: DataMiner (Chrome extension).
- Code: Cheerio (Node.js).
Crawling needs proxies, rate limits—ugh. Only go there if you *have* to.
---
OP REPLY:
Wow, thanks everyone! This makes *so* much more sense now.
I tried BeautifulSoup for scraping prices (followed a YouTube tutorial), and it worked! But yeah, got blocked after 100 requests lol. Guess I need to slow down or use proxies?
Follow-up Q: How do you avoid getting blocked while scraping? Is rotating IPs the only way?
(Also, Dexi.io looks cool—gonna test that next!)
For a beginner, stick to scraping single sites. Tools:
- Browser: DataMiner (Chrome extension).
- Code: Cheerio (Node.js).
Crawling needs proxies, rate limits—ugh. Only go there if you *have* to.
---
OP REPLY:
Wow, thanks everyone! This makes *so* much more sense now.
I tried BeautifulSoup for scraping prices (followed a YouTube tutorial), and it worked! But yeah, got blocked after 100 requests lol. Guess I need to slow down or use proxies?
Follow-up Q: How do you avoid getting blocked while scraping? Is rotating IPs the only way?
(Also, Dexi.io looks cool—gonna test that next!)
