If you’re dealing with a ton of URLs, consider using `aiohttp` for async requests. It’s perfect for python get url ignore 404 because it’s fast and handles errors well.
Here’s a quick example:
```python
import aiohttp
import asyncio
async def fetch(session, url):
try:
async with session.get(url) as response:
if response.status == 404:
print(f"404 for {url}, skipping!")
else:
return await response.text()
except aiohttp.ClientError:
print(f"Error with {url}, moving on!")
async def main():
async with aiohttp.ClientSession() as session:
await fetch(session, "http://example.com/404")
```
This is great for high-performance scraping.
Here’s a quick example:
```python
import aiohttp
import asyncio
async def fetch(session, url):
try:
async with session.get(url) as response:
if response.status == 404:
print(f"404 for {url}, skipping!")
else:
return await response.text()
except aiohttp.ClientError:
print(f"Error with {url}, moving on!")
async def main():
async with aiohttp.ClientSession() as session:
await fetch(session, "http://example.com/404")
```
This is great for high-performance scraping.
