Used crawlee for a client project last month. It’s *good*, but not perfect.
Pros:
- Super easy to get running.
- Handles anti-bot stuff better than Puppeteer.
Cons:
- Memory usage spikes on big crawls.
- Playwright’s still better for super interactive sites.
If you’re already in Node.js land, it’s worth a shot. Otherwise, Scrapy’s probably safer.
Pros:
- Super easy to get running.
- Handles anti-bot stuff better than Puppeteer.
Cons:
- Memory usage spikes on big crawls.
- Playwright’s still better for super interactive sites.
If you’re already in Node.js land, it’s worth a shot. Otherwise, Scrapy’s probably safer.