How to Efficiently Handle Splitting into Batches in Sklearn for Large Datasets?

20 Replies, 2052 Views

Hey! Splitting into batches sklearn can be a pain with big data. Have you tried `KFold` with `shuffle=True`? It can help with balancing.

Also, check out `Modin`—it’s a drop-in replacement for pandas and handles big data way better.

And yeah, avoid loading everything at once. Use `Pandas’ chunksize` or switch to a database like PostgreSQL.

Messages In This Thread



Users browsing this thread: 1 Guest(s)