Wow, thanks everyone for the awesome suggestions! I tried the try-except block with `requests.get()` and it worked perfectly for my python get url ignore 404 issue. I also checked out `httpx` and it looks super promising for my next project.
Quick question though—has anyone used `Scrapy` for large-scale scraping? I’m curious if it’s worth the learning curve. Also, thanks for the `aiohttp` tip—I’ll definitely give that a shot for async stuff. You all rock! 🙌
Quick question though—has anyone used `Scrapy` for large-scale scraping? I’m curious if it’s worth the learning curve. Also, thanks for the `aiohttp` tip—I’ll definitely give that a shot for async stuff. You all rock! 🙌