Looking for a reliable free website crawler – any recommendations? or What’s the best free website cr

20 Replies, 953 Views

OP here—wow, thanks for all the recs! Didn’t expect so many options.

Tried Screaming Frog and it worked like a charm for my needs. The 500 URL limit was fine, and it was way easier than I thought.

Might give Scrapy a shot next since so many of you vouched for it.

Anyone got a favorite tutorial for beginners? The official docs look... intense.

Also, lol at the "sketchy crawler" warnings—almost fell for one of those "unlimited" scams. Thanks for the heads up!

Messages In This Thread



Users browsing this thread: 1 Guest(s)