Wow, thanks so much for all the tips, everyone! I didn’t expect so many helpful replies. I tried breaking the script into smaller chunks like some of you suggested, and it’s already running way smoother.
I also cleared the cache and switched to “Run on Cloud,” which seems to be more stable. Still getting a few errors here and there, but it’s way better than before.
I’ll definitely check out Octoparse and Scrapy as alternatives. And thanks for the heads-up about the API limits—I had no idea that could be causing some of the issues.
Really appreciate all the advice! You guys are awesome.
I also cleared the cache and switched to “Run on Cloud,” which seems to be more stable. Still getting a few errors here and there, but it’s way better than before.
I’ll definitely check out Octoparse and Scrapy as alternatives. And thanks for the heads-up about the API limits—I had no idea that could be causing some of the issues.
Really appreciate all the advice! You guys are awesome.