For a universal tool, check out `databricks`. It handles all those formats and scales like crazy.
But if you’re on a budget, `csvsql` (from csvkit) lets you query CSVs with SQL. Super handy for quick fixes.
Edge cases? I just yell at my screen and then write a regex. Works 60% of the time, every time.
---
Wow, thanks for all the awesome replies!
I’m definitely gonna try pandas + pyarrow—sounds like a game-changer. And OpenRefine looks perfect for cleaning up the messier files.
Anyone have thoughts on using `polars` instead of pandas for faster parsing data? Heard it’s lightning-fast but not sure if it’s worth the switch.
Also, the CSV header-on-row-5 horror story? That’s brutal. I’ve been there with Excel files where the "data" starts on like, tab 3. Why do people do this to us? 😂
Thanks again, y’all!
But if you’re on a budget, `csvsql` (from csvkit) lets you query CSVs with SQL. Super handy for quick fixes.
Edge cases? I just yell at my screen and then write a regex. Works 60% of the time, every time.
---
Wow, thanks for all the awesome replies!
I’m definitely gonna try pandas + pyarrow—sounds like a game-changer. And OpenRefine looks perfect for cleaning up the messier files.
Anyone have thoughts on using `polars` instead of pandas for faster parsing data? Heard it’s lightning-fast but not sure if it’s worth the switch.
Also, the CSV header-on-row-5 horror story? That’s brutal. I’ve been there with Excel files where the "data" starts on like, tab 3. Why do people do this to us? 😂
Thanks again, y’all!
