Ensemble Model not as expected

[tomneo2004]

In the video @mrdbourke able to use ensemble model to beat model_1. However, it is not the case for me.

The result of mean from ensemble model's prediction
{'mae': 582.7927, 'mape': 2.6410944, 'mase': 1.0238054, 'mse': 1183840.5, 'rmse': 1088.0443}

The result of median from ensemble model's prediction
{'mae': 577.1618, 'mape': 2.6071944, 'mase': 1.0139134, 'mse': 1177605.8, 'rmse': 1085.1755}

And the result of model_1
{'mae': 568.95123, 'mape': 2.5448983, 'mase': 0.9994897, 'mse': 1171744.0, 'rmse': 1082.4713}

Naive still the best
{'mae': 567.9802, 'mape': 2.516525, 'mase': 0.99957, 'mse': 1147547.0, 'rmse': 1071.2362}

What could be the issue behind this?

The ensemble model

def get_ensemble_models(horizon=HORIZON,
                        train_data=train_dataset,
                        test_data=test_dataset,
                        num_iter=10,
                        num_epochs=100,
                        loss_fns=["mae", "mse", "mape"]):
  """
  Returns a list of num_iter models each trained on MAE, MSE and MAPE loss.

  For example, if num_iter=10, a list of 30 trained models will be returned:
  10 * len(["mae", "mse", "mape"])
  """
  # make empty list for trained ensemble models
  ensemble_models = []

  # create num_iter number of models per loss function
  for i in range(num_iter):
    # build and fit new model with a differnt loss function
    for loss_function in loss_fns:
      print(f"Optimizing model by reducing {loss_function} for {num_epochs} epochs, model number: {i}")

      # construct a simple model (similar to model_1)
      model = tf.keras.Sequential([
                                   layers.Dense(128, kernel_initializer="he_normal", activation="relu"),
                                   layers.Dense(128, kernel_initializer="he_normal", activation="relu"),
                                   layers.Dense(HORIZON)
      ])

      # compile simple model with current loss function
      model.compile(loss=loss_function,
                    optimizer=tf.keras.optimizers.Adam(),
                    metrics=["mae", "mse"])
      
      # fit the current model
      model.fit(train_data,
                epochs=num_epochs,
                verbose=0,
                validation_data=test_data,
                callbacks=[tf.keras.callbacks.EarlyStopping(monitor="val_loss",
                                                            patience=200,
                                                            restore_best_weights=True),
                           tf.keras.callbacks.ReduceLROnPlateau(monitor="val_loss",
                                                                patience=100,
                                                                verbose=1)])
      
      # append fitted model to list of ensemble models
      ensemble_models.append(model)
  
  return ensemble_models

Train ensemble model

%%time
tf.random.set_seed(42)
ensemble_models = get_ensemble_models(num_iter=5,
                                      num_epochs=100)

Function for prediction

# Create a function for ensemble prediction
def get_ensemble_predictions(ensemble_models, data):
  ensemble_preds = []
  for model in ensemble_models:
    y_pred = model.predict(data)
    ensemble_preds.append(y_pred)
  return tf.constant(tf.squeeze(ensemble_preds))

Make predictions

%%time
model_8_pred = get_ensemble_predictions(ensemble_models, test_dataset)
model_8_pred

Predcitions for mean and median

# Aggregate predictions
model_8_pred_mean = tf.reduce_mean(model_8_pred, axis=0)
model_8_pred_median = np.median(model_8_pred, axis=0)

[mrdbourke]

Hey @tomneo2004,

Due the inherit randomness in machine learning algorithms, you should expect to see slightly different results to the video.

Your model not beating the others is a little strange though.

Did you try running the code notebook from the course? (here: https://github.com/mrdbourke/tensorflow-deep-learning/blob/main/10_time_series_forecasting_in_tensorflow.ipynb)

If that notebook still doesn't beat it, it's just another example of how strong the naive model is and how poor machine learning models can be at predicting open market data.