How to Efficiently Handle Splitting into Batches in Sklearn for Large Datasets?

20 Replies, 2060 Views

Hey there! For splitting into batches sklearn, I’d recommend using `train_test_split` with smaller chunks first. If that’s still too much, try `KFold` with `n_splits` set to a higher number.

Another option is `HDF5` for storing your data. It’s super efficient and lets you access parts of the dataset without loading everything into memory.

Also, consider cloud solutions like Google Colab or AWS if your laptop can’t handle it.

Messages In This Thread



Users browsing this thread: 1 Guest(s)