How to Efficiently Handle Splitting into Batches in Sklearn for Large Datasets?

How to Efficiently Handle Splitting into Batches in Sklearn for Large Datasets? - Printable Version

+- Proxy Community (https://proxycommunity.com/forum)
+-- Forum: Use Case (https://proxycommunity.com/forum/forum-use-case)
+--- Forum: Others (https://proxycommunity.com/forum/forum-others)
+--- Thread: How to Efficiently Handle Splitting into Batches in Sklearn for Large Datasets? (/thread-how-to-efficiently-handle-splitting-into-batches-in-sklearn-for-large-datasets)

Pages: 1 2 3

“” - fastDriftX - 22-03-2025

Hey! For splitting into batches sklearn, you might wanna look into `Pipeline` with `PartialFit`. It’s designed for incremental learning and works great with big data.

Also, `KFold` is cool, but if your data’s too big, maybe try `GroupKFold` if you have groups in your data.

And yeah, definitely avoid loading everything into memory. Try `PyTables` or `Feather` for efficient storage.