For reading pd dataframe from local files efficiently, here’s a pro tip:
Pre-allocate memory by specifying `dtype` for each column. Pandas wastes time inferring types otherwise.
Also, `pd.read_feather()` is insanely fast for medium-sized data—way quicker than CSV/Excel.
Downside? Feather’s not as widely supported, but great for intermediate storage.
Pre-allocate memory by specifying `dtype` for each column. Pandas wastes time inferring types otherwise.
Also, `pd.read_feather()` is insanely fast for medium-sized data—way quicker than CSV/Excel.
Downside? Feather’s not as widely supported, but great for intermediate storage.
