Fun fact: You can combine both!
First crawl to find pages (like with Scrapy), then scrape the good stuff.
For big data, check out Apache Storm—overkill for most, but cool if you’re into distributed systems.
First crawl to find pages (like with Scrapy), then scrape the good stuff.
For big data, check out Apache Storm—overkill for most, but cool if you’re into distributed systems.
