What's the best way to build a web crawler in Python from scratch? or How can I optimize my web crawl

22 Replies, 1203 Views

For a web crawler python newbie, start simple!

Use `requests-html`—it’s like requests but handles JS. Not perfect, but easier than Selenium.

```python
from requests_html import HTMLSession
session = HTMLSession()
r = session.get('https://example.com')
r.html.render() # Executes JS
```

Messages In This Thread



Users browsing this thread: 1 Guest(s)