For big XML, `iterparse()` + `clear()` is the golden combo. Otherwise, memory goes brrrr.
lxml Python’s `html` module is lenient, but `etree` is *strict* and faster. Pick your poison.
Also, `lxml.html.clean.Cleaner()` is great for sanitizing scraped HTML.
---
Wow, thanks for all the killer tips! I tried `iterparse()` with `clear()` on a giant XML dump, and it’s *way* better than loading it all at once.
One follow-up: Anyone got a trick for handling malformed HTML that still lets lxml Python stay fast? The `html` module helps, but sometimes it’s still a crawl.
Also, big shoutout for the `remove_blank_text` tip—saved me a headache!
lxml Python’s `html` module is lenient, but `etree` is *strict* and faster. Pick your poison.
Also, `lxml.html.clean.Cleaner()` is great for sanitizing scraped HTML.
---
Wow, thanks for all the killer tips! I tried `iterparse()` with `clear()` on a giant XML dump, and it’s *way* better than loading it all at once.
One follow-up: Anyone got a trick for handling malformed HTML that still lets lxml Python stay fast? The `html` module helps, but sometimes it’s still a crawl.
Also, big shoutout for the `remove_blank_text` tip—saved me a headache!
