Here’s a forum post option for you—casual, opinionated, and human-like:
---
"Has anyone successfully managed to scrape data using LLM? How did you do it?"
Hey folks,
So I’ve been messing around with the idea to scrape data using LLM instead of the usual bs4 or selenium stuff.
Anyone actually pulled this off? Like, did you just prompt the LLM to extract stuff from a webpage or feed it raw HTML?
I tried a few things, but tbh, it feels hit or miss. Sometimes it’s *weirdly* good at pulling structured data, other times it just hallucinates nonsense.
Tools? Prompts? Workarounds? Or is this just a pipe dream lol.
Also, how does it compare to traditional scraping? Faster? More reliable? Or just a shiny distraction?
---
Let me know if you want it tweaked! Shortened, more typos, etc.
Yeah, I’ve tried to scrape data using llm a few times, and it’s… interesting.
I fed raw HTML into GPT-4 and asked it to extract specific tables or lists. Worked okay for simple stuff, but yeah, the hallucinations are real.
For better results, I pre-processed the HTML with something like BeautifulSoup to clean it up first. Less noise = better accuracy.
Still, traditional scraping is way more reliable for bulk work. LLMs are fun for quick, one-off extractions though.
Honestly, scraping with LLMs feels like using a flamethrower to light a candle.
It *can* work, but why? Tools like Scrapy or Puppeteer are built for this.
That said, if you’re dead set on it, check out LLM-based tools like Diffbot or Browse AI. They kinda bridge the gap between traditional scraping and AI.
But yeah, for most cases, stick to the classics.
I’ve had some luck with Claude + raw HTML!
The trick is to give it *very* clear instructions, like "extract all product names and prices from this HTML table."
Still, it’s slower than traditional methods, and you gotta watch for errors.
For small projects, it’s neat. For anything big? Nah.
Pipe dream? Maybe not, but close.
I tried using LLMs to scrape data from dynamic pages, and it was a mess.
Sometimes it’d nail it, other times it’d invent data that wasn’t even there.
If you’re gonna try, pair it with something like Playwright to get the HTML first. Raw LLM scraping is too unpredictable.
Why not both?
I use traditional scrapers to get the data, then LLMs to clean/parse it. Best of both worlds.
For example, scrape with Selenium, then dump the messy output into GPT-4 to structure it. Works way better than relying on the LLM alone.
LLMs for scraping? Sounds cool until you realize how expensive it gets.
Every API call adds up, and the latency is brutal compared to regex or XPath.
If you’re just playing around, sure. But for real work? Hard pass.
OP here—thanks for all the replies!
Sounds like the consensus is that scrape data using llm is possible but kinda janky for most cases.
I’ll give Diffbot a shot since a few of you mentioned it. Also like the idea of combining traditional scraping with LLMs for cleanup.
One follow-up: anyone tried this with local LLMs (like Llama 3)? Wondering if it’s any better for privacy-heavy projects.
I’ve seen people use OpenAI’s API to extract data from PDFs or messy HTML.
It’s *kinda* reliable if you chunk the input and chain prompts.
But honestly, unless you’re dealing with unstructured garbage, traditional tools are faster and cheaper.
