IMO, the debate around web scraping vs web crawling misses the real issue: *ethics*.
Sure, you *can* scrape anything, but should you? Some sites ban scrapers outright.
Tools like Screaming Frog are great for *light* crawling (SEO stuff). For scraping, maybe start with Import.io.
Just… don’t be *that* guy who crashes a site.
Sure, you *can* scrape anything, but should you? Some sites ban scrapers outright.
Tools like Screaming Frog are great for *light* crawling (SEO stuff). For scraping, maybe start with Import.io.
Just… don’t be *that* guy who crashes a site.
