Wait, so if I’m scraping product prices from Amazon, that’s just scraping, right?
But if I wanna find all product pages first, I’d need to crawl?
Makes sense!
Tools I use:
- Scrapy for crawling (built-in throttling helps avoid bans)
- Puppeteer for JS-heavy sites
Anyone else get rate-limited constantly? How do you bypass that?
But if I wanna find all product pages first, I’d need to crawl?
Makes sense!
Tools I use:
- Scrapy for crawling (built-in throttling helps avoid bans)
- Puppeteer for JS-heavy sites
Anyone else get rate-limited constantly? How do you bypass that?