How to Web Scrape Images from HTML Using BeautifulSoup (bs4) – Any Tips or Best Practices?

20 Replies, 2298 Views

Hey! For handling relative URLs, you can use `urllib.parse.urljoin` to convert them to absolute URLs. Just pass the base URL of the site and the relative URL, and it’ll do the magic.

For lazy-loaded images, yeah, sometimes you need to scroll or use a headless browser like Selenium or Playwright to trigger the loading. It’s a bit more work but worth it if the images are crucial for your project.

Here’s a quick snippet:
```python
from urllib.parse import urljoin
base_url = "https://example.com"
for img in soup.find_all('img'):
src = img.get('src')
if src:
absolute_url = urljoin(base_url, src)
print(absolute_url)
```
Hope this helps!

Messages In This Thread



Users browsing this thread: 1 Guest(s)