How to Efficiently Handle Splitting into Batches in Sklearn for Large Datasets?

20 Replies, 1989 Views

Hey! For splitting into batches sklearn, you might wanna look into `Pipeline` with `PartialFit`. It’s designed for incremental learning and works great with big data.

Also, `KFold` is cool, but if your data’s too big, maybe try `GroupKFold` if you have groups in your data.

And yeah, definitely avoid loading everything into memory. Try `PyTables` or `Feather` for efficient storage.



Users browsing this thread: 1 Guest(s)