Save_model and load_model function don't save/load optimizer state

[ghost]

I was trying to save and load a ConditionalLogit model for resuming a training, but the implementation inside the base class ChoiceModel doesn't save and load the optimizer state, which it's essential when resuming a training.

[VincentAuriau]

Let me check if I can quickly come up with a solution for this . I will update you soon.

For the time being I suppose you can re instantiate the optimizer - you may lose a few epochs but I suppose the results should be ok.

[VincentAuriau]

1. Yes you can ignore the message (and I removed it)
2. just commited this - I suppose I will later add a warning message that the saved model is overwritten
3. Is it a numpy error that is raised ? Can you copy the exact error please?

And thanks a lot for your feedbacks !

[ghost]

3. Sure. Here it is:

config['learning_rate'] it's simply a float value in my case:

I simply commented the lines where it converts .tolist() and now it works.

[ghost]

Here is simple example with a callback to save a model every n epochs.

If you want fancier options you can check the official implementation from tensorflow and copy some stuff

This works perfectly ;) I managed to save the checkpoints. But one question is when I want to resume and use e.g. an model from epoch 8, shouldn't the epoch number in the fit() function also be resumed starting from the one that was saved (line 289 in base_model.py - the tqdm,trange starts at position 0 rather than resuming from the epoch it was left). I'm attaching here an example:

I would've expected that the epochs start from epoch 9 and continue until epoch 15

[VincentAuriau]

I pushed a fix - it looks like depending on the tf version, the configs returned have different types - let me know if it still does not work

[VincentAuriau]

Here is simple example with a callback to save a model every n epochs.
If you want fancier options you can check the official implementation from tensorflow and copy some stuff

This works perfectly ;) I managed to save the checkpoints. But one question is when I want to resume and use e.g. an model from epoch 8, shouldn't the epoch number in the fit() function also be resumed starting from the one that was saved (line 289 in base_model.py - the tqdm,trange starts at position 0 rather than resuming from the epoch it was left). I'm attaching here an example: I would've expected that the epochs start from epoch 9 and continue until epoch 15

It is a question I had. Usually the number of epochs is given directly in the fit. For simplicity I added it in the __init__ so that it can be saved as an hyper parameter. I kinda kept this behavior so that the model wan be trained several times if needs be (without having to change the n_epochs argument).

I can add an argument 'total_trained_epochs' or something similar that might help you know how many additional epochs you need.