Crawlee is def a step up from older tools, especially for JS sites. But if you're dealing with anti-bot systems, you might wanna pair it with something like Bright Data’s scraper.
I used it for a mid-sized project, and the proxy rotation saved me a ton of headaches. Docs are meh tho—had to dig through GitHub issues a few times.
Also, the name? No clue, but it’s grown on me. Maybe ‘cause it crawls like a lil’ bug?
I used it for a mid-sized project, and the proxy rotation saved me a ton of headaches. Docs are meh tho—had to dig through GitHub issues a few times.
Also, the name? No clue, but it’s grown on me. Maybe ‘cause it crawls like a lil’ bug?