What's the most efficient method for parsing data from different file formats? or How do you handle e

16 Replies, 1692 Views

For a universal tool, check out `databricks`. It handles all those formats and scales like crazy.

But if you’re on a budget, `csvsql` (from csvkit) lets you query CSVs with SQL. Super handy for quick fixes.

Edge cases? I just yell at my screen and then write a regex. Works 60% of the time, every time.

---

Wow, thanks for all the awesome replies!

I’m definitely gonna try pandas + pyarrow—sounds like a game-changer. And OpenRefine looks perfect for cleaning up the messier files.

Anyone have thoughts on using `polars` instead of pandas for faster parsing data? Heard it’s lightning-fast but not sure if it’s worth the switch.

Also, the CSV header-on-row-5 horror story? That’s brutal. I’ve been there with Excel files where the "data" starts on like, tab 3. Why do people do this to us? 😂

Thanks again, y’all!

Messages In This Thread



Users browsing this thread: 1 Guest(s)