What to do if the data is too big to fit into memory?

[mertbakir]

Hello, how do we handle data that is big enough to fit into memory? I need to train the model in batches.

[jmoralez]

Hey. All training data needs to be in memory, if it doesn't fit you can try using a remote cluster and the distributed interface (guide).

[mertbakir]

Thanks, I thought we can do incremental updates on the model iteratevily while training. So, you are saying there is no way?

[jmoralez]

There may be a way but certainly it won't be easy at all, I suggest the cluster approach if you're sure that using more data will improve the model.