About examples/trials/mnist-keras.py

[sayakpaul]

Referring to this example: https://github.com/microsoft/nni/tree/master/examples/trials/mnist-keras.

So, I ran it the mnist-keras.py script and here's the log:

Epoch 1/10
  1/300 [..............................] - ETA: 1:16:52 - loss: 2.3189 - accuracy: 0.0700WARNING:tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0035s vs `on_train_batch_end` time: 0.0042s). Check your callbacks.
[2021-07-12 05:08:47] WARNING (tensorflow/MainThread) Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0035s vs `on_train_batch_end` time: 0.0042s). Check your callbacks.
300/300 [==============================] - 18s 9ms/step - loss: 0.2102 - accuracy: 0.9388 - val_loss: 0.0574 - val_accuracy: 0.9816
[2021-07-12 05:08:49] INFO (nni/MainThread) Intermediate result: 0.9815999865531921  (Index 0)
Epoch 2/10
300/300 [==============================] - 2s 8ms/step - loss: 0.0506 - accuracy: 0.9846 - val_loss: 0.0419 - val_accuracy: 0.9865
[2021-07-12 05:08:52] INFO (nni/MainThread) Intermediate result: 0.9865000247955322  (Index 1)
Epoch 3/10
300/300 [==============================] - 2s 8ms/step - loss: 0.0334 - accuracy: 0.9899 - val_loss: 0.0377 - val_accuracy: 0.9879
[2021-07-12 05:08:54] INFO (nni/MainThread) Intermediate result: 0.9879000186920166  (Index 2)
Epoch 4/10
300/300 [==============================] - 2s 8ms/step - loss: 0.0227 - accuracy: 0.9928 - val_loss: 0.0352 - val_accuracy: 0.9893
[2021-07-12 05:08:56] INFO (nni/MainThread) Intermediate result: 0.989300012588501  (Index 3)
Epoch 5/10
300/300 [==============================] - 2s 8ms/step - loss: 0.0163 - accuracy: 0.9948 - val_loss: 0.0303 - val_accuracy: 0.9908
[2021-07-12 05:08:59] INFO (nni/MainThread) Intermediate result: 0.9908000230789185  (Index 4)
Epoch 6/10
300/300 [==============================] - 2s 8ms/step - loss: 0.0126 - accuracy: 0.9958 - val_loss: 0.0365 - val_accuracy: 0.9888
[2021-07-12 05:09:01] INFO (nni/MainThread) Intermediate result: 0.9887999892234802  (Index 5)
Epoch 7/10
300/300 [==============================] - 2s 8ms/step - loss: 0.0093 - accuracy: 0.9971 - val_loss: 0.0398 - val_accuracy: 0.9896
[2021-07-12 05:09:04] INFO (nni/MainThread) Intermediate result: 0.9896000027656555  (Index 6)
Epoch 8/10
300/300 [==============================] - 2s 8ms/step - loss: 0.0070 - accuracy: 0.9976 - val_loss: 0.0413 - val_accuracy: 0.9886
[2021-07-12 05:09:06] INFO (nni/MainThread) Intermediate result: 0.9886000156402588  (Index 7)
Epoch 9/10
300/300 [==============================] - 2s 8ms/step - loss: 0.0061 - accuracy: 0.9980 - val_loss: 0.0396 - val_accuracy: 0.9897
[2021-07-12 05:09:09] INFO (nni/MainThread) Intermediate result: 0.9897000193595886  (Index 8)
Epoch 10/10
300/300 [==============================] - 2s 8ms/step - loss: 0.0059 - accuracy: 0.9981 - val_loss: 0.0424 - val_accuracy: 0.9881
[2021-07-12 05:09:11] INFO (nni/MainThread) Intermediate result: 0.988099992275238  (Index 9)
[2021-07-12 05:09:13] INFO (nni/MainThread) Final result: 0.988099992275238

I don't think there is any trial going on behind hood for this one. How should I configure a trail for this?

[scarlett2018]

@sayakpaul -
An example python code can be started directly as usually Python codes, but if you would like to leverage NNI's capability, you need to install NNI and set up configurations for this training codes. please refer to the Quick Start for example.

[sayakpaul]

I am not really sure if that answers the question. I believe the example I mentioned already leverages NNI's capabilities otherwise this part of the output won't have come:

[2021-07-12 05:09:11] INFO (nni/MainThread) Intermediate result: 0.988099992275238  (Index 9)
[2021-07-12 05:09:13] INFO (nni/MainThread) Final result: 0.988099992275238

Also, I would like to reiterate that I am referring to the official example here: https://github.com/microsoft/nni/tree/master/examples/trials/mnist-keras. This already has NNI incorporated I believe.

[jonsnows]

U can check the log on tensorboard, the tfevents in the directory 'NNI_OUTPUT_DIR‘ that you have set.

[sayakpaul]

The tutorial does not mention it anywhere. Can you append your observations in a notebook as I think that might be helpful.