Man, I wish I’d asked this when I started. For how to setup web scrapper, here’s my take:
Use BeautifulSoup. Scrapy’s cool but has a steep curve.
Also, use `selenium` ONLY if you *must* scrape JS-heavy sites. It’s slow but works.
Biggest rookie mistake? Not saving data as you go. Lost hours of scraping cuz my script crashed.
Use BeautifulSoup. Scrapy’s cool but has a steep curve.
Also, use `selenium` ONLY if you *must* scrape JS-heavy sites. It’s slow but works.
Biggest rookie mistake? Not saving data as you go. Lost hours of scraping cuz my script crashed.