What's the best way to read pd dataframe from local in Python? or How do I read pd dataframe from loc

16 Replies, 1560 Views

Don’t sleep on `pd.read_sql()` if your data’s in a local SQLite DB!

Even for CSV/Excel, dumping to SQLite first can make querying faster later.

For Excel, `pd.ExcelFile()` lets you read sheets without reloading the file each time.

Also, `memory_map=True` in `pd.read_csv()` can help with huge files.

---

Wow, thanks for all the tips!

I tried `pd.read_parquet()` and it’s *way* faster for my big datasets. Also, the `usecols` trick saved me so much time.

One follow-up—anyone know if `modin` works well with `chunksize`? Or is it better to just let it parallelize the whole load?

Thanks again, y’all are legends!

Messages In This Thread



Users browsing this thread: 1 Guest(s)