Wow, thanks for all the replies! This makes way more sense now.
I tried scraping with BeautifulSoup for a static site, and it worked perfectly. But I see now why crawling would be needed for dynamic stuff.
Follow-up Q: How do you handle sites that block scrapers? Proxies? Or just respect robots.txt and go slow?
(Also, gonna check out Scrapy for my next project—sounds powerful!)
I tried scraping with BeautifulSoup for a static site, and it worked perfectly. But I see now why crawling would be needed for dynamic stuff.
Follow-up Q: How do you handle sites that block scrapers? Proxies? Or just respect robots.txt and go slow?
(Also, gonna check out Scrapy for my next project—sounds powerful!)
