[b]"What's the best way to build a web crawler in Python from scratch?"[/b] or [b]"How can I optimize my web crawl

22 Replies, 704 Views

JS-heavy sites suck for web crawler python scripts. Sadly, requests won’t cut it.

Playwright’s lighter than Selenium and works great. Example:
```python
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto("https://example.com")
print(page.content())
```

Messages In This Thread



Users browsing this thread: 1 Guest(s)