Don’t sleep on `pd.read_sql()` if your data’s in a local SQLite DB!
Even for CSV/Excel, dumping to SQLite first can make querying faster later.
For Excel, `pd.ExcelFile()` lets you read sheets without reloading the file each time.
Also, `memory_map=True` in `pd.read_csv()` can help with huge files.
---
Wow, thanks for all the tips!
I tried `pd.read_parquet()` and it’s *way* faster for my big datasets. Also, the `usecols` trick saved me so much time.
One follow-up—anyone know if `modin` works well with `chunksize`? Or is it better to just let it parallelize the whole load?
Thanks again, y’all are legends!
Even for CSV/Excel, dumping to SQLite first can make querying faster later.
For Excel, `pd.ExcelFile()` lets you read sheets without reloading the file each time.
Also, `memory_map=True` in `pd.read_csv()` can help with huge files.
---
Wow, thanks for all the tips!
I tried `pd.read_parquet()` and it’s *way* faster for my big datasets. Also, the `usecols` trick saved me so much time.
One follow-up—anyone know if `modin` works well with `chunksize`? Or is it better to just let it parallelize the whole load?
Thanks again, y’all are legends!
