What's the Difference Between Web Crawling vs Scraping? Need Help Understanding! or Web Crawling vs S

24 Replies, 1272 Views

lol i was confused about web crawling vs scraping too until i had to do it for work.

crawling = finding pages (like googlebot)
scraping = taking data from those pages

if u know the exact pages u need, just scrape. no need to crawl. but if u dont, u gotta crawl first.

tools? scrapy is OP for both. for easy stuff, try octoparse or parsehub.

watch out for legal stuff tho. some sites block scrapers. use proxies if u gotta.

Messages In This Thread



Users browsing this thread: 1 Guest(s)