"Has anyone had success with training an LLM using books? Tips welcome!"
Hey folks,
So I’ve been digging into training an LLM using books—classic novels, textbooks, you name it. Seems like a goldmine for high-quality text, but I’m kinda stuck on the best way to prep the data.
Anyone here actually pulled this off? How’d you handle formatting issues (like footnotes or weird OCR errors)? And is it worth the effort compared to just scraping the web?
Also, did you chunk the text by chapters or just throw it all in raw? Low-key worried about overfitting if the model gets too cozy with one author’s style lol.
Would love to hear your war stories or any gotchas!
Cheers.
Hey! I tried training an LLM using books last year, and it was a mixed bag. The biggest headache was cleaning up the text—OCR errors are brutal, especially with older books.
I used tools like Tesseract for OCR and then cleaned up the output with regex in Python. For chunking, I split by chapters to keep some structure.
Honestly, web scraping might be easier, but books give you way better quality if you’re willing to put in the work. Check out Project Gutenberg for clean, pre-formatted classics!
Training an LLM using books? Yeah, it’s doable but messy. Footnotes and weird formatting are the worst.
I ended up using pdftotext for PDFs and then wrote a custom script to strip out footnotes. For chunking, I went with paragraphs—keeps things digestible for the model.
Pro tip: Mix in some web data to avoid overfitting. Otherwise, your model might start sounding like Dickens lol.
I’ve had some success with training an LLM using books, but it’s not plug-and-play. The key is preprocessing.
Tools like Apache Tika helped me extract text cleanly, and I used spaCy for splitting chapters. Also, don’t forget to deduplicate—some books have repeating intros or copyright stuff.
Web data’s faster, but books give you depth. Worth it if you’re aiming for a niche model.
Oh man, training an LLM using books is a grind. OCR errors are the devil.
I found that Google Books’ scans are cleaner than most, but you still need to post-process. For chunking, I did sentences—easier for the model to handle.
Also, watch out for copyright if you’re using newer books. Stick to public domain unless you’re feeling risky.
Training an LLM using books is totally worth it if you want rich, nuanced text. But yeah, the prep work sucks.
I used Calibre to convert eBooks to plain text, then cleaned them up with OpenRefine. For chunking, I went with 512-token blocks—seemed to work well for fine-tuning.
Web data’s noisy, but books? Pure gold. Just be ready for some manual labor.
If you’re training an LLM using books, don’t skip the cleanup step. I learned that the hard way.
Tools like Textract (AWS) saved me from OCR hell. For chunking, I mixed it up—some chapters, some random chunks. Helps avoid overfitting.
Also, throw in some modern text so your model doesn’t sound like it’s from the 1800s.
Training an LLM using books is a solid move, but it’s not for the faint-hearted.
I used PyPDF2 for PDFs and then ran the text through LangChain’s splitter. Worked pretty well!
Biggest gotcha? Some books have hidden formatting (like italics as underscores). Took me ages to fix that.
Hey, thanks for all the tips, everyone! This is super helpful.
I tried preprocessing a few books with Tesseract and regex like some of you suggested, and it’s already way cleaner. Still running into issues with footnotes, though—anyone have a magic script for that?
Also, love the idea of mixing in web data to avoid overfitting. Gonna give that a shot next.
Cheers!
