How to Efficiently Handle Splitting into Batches in Sklearn for Large Datasets?

20 Replies, 2482 Views

Dude, I feel you. Splitting into batches sklearn can be a pain with huge data. Have you tried `Dask-ML`? It’s built for big datasets and integrates well with sklearn.

Also, `KFold` is good, but for really big data, maybe try `TimeSeriesSplit` if your data has a time component.

And yeah, avoid loading everything at once. Use `Pandas’ chunksize` or switch to a database like SQLite.

Messages In This Thread



Users browsing this thread: 1 Guest(s)