Skip to content

[ML] Incorrect hyperparameter importance computation #1859

Open
@valeriy42

Description

@valeriy42

There are cases when we output 0 absolute importance and non-0 relative importance for estimated hyperparameters. I particularly noticed it on the iris dataset from QA:

configuration:

"source" : {
            "query" : {
              "match_all" : { }
            },
            "index" : [
              "iris"
            ]
          },
          "dest" : {
            "index" : "dest_iris_random_seed_80_1618920715038",
            "results_field" : "ml"
          },
          "analysis" : {
            "classification" : {
              "early_stopping_enabled" : true,
              "randomize_seed" : -2898491962519781504,
              "dependent_variable" : "class",
              "num_top_classes" : 2,
              "training_percent" : 80.0,
              "class_assignment_objective" : "maximize_minimum_recall",
              "prediction_field_name" : "class_prediction"
            }
          },

results

"hyperparameters" : [
          {
            "name" : "downsample_factor",
            "value" : 0.6934178644572064,
            "absolute_importance" : 0.0,
            "relative_importance" : 1.0118878992805921E-4,
            "supplied" : false
          },
          {
            "name" : "alpha",
            "value" : 1.2624949147815578,
            "absolute_importance" : 0.0,
            "relative_importance" : 0.001724798926887357,
            "supplied" : false
          },
          {
            "name" : "lambda",
            "value" : 0.22289001486228566,
            "absolute_importance" : 0.0,
            "relative_importance" : 0.0701837648621131,
            "supplied" : false
          },
          {
            "name" : "gamma",
            "value" : 0.9944221836929659,
            "absolute_importance" : 0.0,
            "relative_importance" : 0.018077482192720387,
            "supplied" : false
          },
          {
            "name" : "soft_tree_depth_limit",
            "value" : 5.203599644200018,
            "absolute_importance" : 0.0,
            "relative_importance" : 0.3711418247303484,
            "supplied" : false
          },
          {
            "name" : "soft_tree_depth_tolerance",
            "value" : 0.07500000000000001,
            "absolute_importance" : 0.0,
            "relative_importance" : 0.0013927082506636976,
            "supplied" : false
          },
          {
            "name" : "eta",
            "value" : 0.37606030930863943,
            "absolute_importance" : 0.0,
            "relative_importance" : 0.005923992129374061,
            "supplied" : false
          },
          {
            "name" : "eta_growth_rate_per_tree",
            "value" : 1.307459346906221,
            "absolute_importance" : 0.0,
            "relative_importance" : 0.038073625498778164,
            "supplied" : false
          },
          {
            "name" : "feature_bag_fraction",
            "value" : 0.7125,
            "absolute_importance" : 0.0,
            "relative_importance" : 3.0057289583808306E-4,
            "supplied" : false
          }
        ]

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions