AutoML Forecast best_trial useage

[ssatti0]

I've run optuna to tune the features in AutoMLForecast where I have models=[list of many ml models], and using the training data only. The data consists of different unique_ids.
Is there a way to use the auto_mlf.best_trial in the training of the AutoML models with all available data? I need this in order to be able to forecast into the future.
Thanks,

[jmoralez]

Hey. The best parameters are used to train a final model on all of the data, so after the optimization process you can just use predict to forecast into the future.

[ssatti0]

Thank you for the quick response @jmoralez, appreciated.
I need to see/calculate the error metrics for the optuna run, by model for each of the unique_ids. I'm particularly interesed in MAE, RMSE and MAPE. Is there a way to get those?

[jmoralez]

Only the cross validation loss is saved as the trial's value. If you've already split your data into train and valid you can just predict when the process finishes and evaluate that using your validation data. If you want to perform cross validation again you can do that for each model, e.g.

for name, model in auto_mlf.models_.items():
    cv_res = model.cross_validation(...)
    evaluate(cv_res, ...)