Retrieve Best Model Using HyperDriveStepRun Within Pipeline

[julianaddison]

I'm facing an issue with HyperDriveStepRun. Have been using HyperDriveStepRun to retrieve the best model in a pipeline. However it's recently been throwing a Service invocation timed out error. Somewhat puzzled as the same pipeline was working in previous runs. The last I ran the pipeline was in Mar 2021 so was expecting it to work. Would you have any idea what the error means?

The pipeline:

Step 3, hyperdrive_save_best_model.py

The error is thrown in Step 3 at line
step_run = HyperDriveStepRun(step_run=pipeline_run.find_step_run('model_training_with_hyperdrive')[0])

I'm able to find the step in the pipeline by running pipeline_run.find_step_run('model_training_with_hyperdrive')[0].get_details() which returns the output below

{'runId': '81afda1d-e09a-4b66-9916-a70d31fe5aaf', 'status': 'Completed', 'startTimeUtc': '2021-04-26T10:04:55.388077Z', 'endTimeUtc': '2021-04-26T12:02:13.912549Z', 'properties': {'ContentSnapshotId': 'f60b2327-8e65-4245-ae9d-ad1d15052db9', 'StepType': 'HyperDriveStep', 'ComputeTargetType': 'HyperDrive', 'azureml.moduleid': '1997d65f-e755-4b48-b7b6-802cebe0e932', 'azureml.runsource': 'azureml.StepRun', 'azureml.nodeid': '1b54a211', 'azureml.pipelinerunid': '4305e2b4-5695-4736-b081-6710b3e7cb64'}, 'inputDatasets': [], 'logFiles': {'logs/azureml/executionlogs.txt': 'https://cerespalmml0992777240.blob.core.windows.net/azureml/ExperimentRun/dcid.81afda1d-e09a-4b66-9916-a70d31fe5aaf/logs/azureml/executionlogs.txt?sv=2019-02-02&sr=b&sig=fQHeUuUTSKkh7RXUMDIr59sqvwXhzWVPBNXp4vaQ%2F%2F4%3D&st=2021-04-26T11%3A57%3A26Z&se=2021-04-26T20%3A07%3A26Z&sp=r', 'logs/azureml/stderrlogs.txt': 'https://cerespalmml0992777240.blob.core.windows.net/azureml/ExperimentRun/dcid.81afda1d-e09a-4b66-9916-a70d31fe5aaf/logs/azureml/stderrlogs.txt?sv=2019-02-02&sr=b&sig=Re57T4Nx1MERInO8xXTGQ7gY%2BFmxfpQKklTPSbV40ig%3D&st=2021-04-26T11%3A57%3A26Z&se=2021-04-26T20%3A07%3A26Z&sp=r', 'logs/azureml/stdoutlogs.txt': 'https://cerespalmml0992777240.blob.core.windows.net/azureml/ExperimentRun/dcid.81afda1d-e09a-4b66-9916-a70d31fe5aaf/logs/azureml/stdoutlogs.txt?sv=2019-02-02&sr=b&sig=cQ5eMV8tG%2BsmJDAd9AYvtNkvVssq3YdcP4CGIsQn6MA%3D&st=2021-04-26T11%3A57%3A26Z&se=2021-04-26T20%3A07%3A26Z&sp=r'}}

Originally posted by @julianaddison in #269 (comment)

[julianaddison]

Any updates on this issue?

[shbijlan]

@julianaddison I have forwarded this issue to engineering. Will update you after investigation.

[sonnypark]

Hi @julianaddison,
This is more like related to how HyperDriveStepRun gets HyderDriveRun. The error shows it had issue while it was getting run details on the experiment. The reason you didn't see same issue to get the step run was such operation only gets the specific run from the exp while the HyperDriveStepRun was accessing all the runs in the exp. So either you had some runs the script couldn't access access or just possibly transient issue.
If you see such issue persistently, here is workarounds you can use.

1. create a new Experiment so that you can minimize such errors.
2. use HyperDriveRun directly. You can get the runid of the hyperdrive run from the Hyperdrive step run using get_children.

Please let me know if you have any further questions.

[julianaddison]

Hi @sonnypark,

Somewhat confused here. I'm interested in finding the best model across all runs in the hyperdrive step which is why i'm using HyperDriveStepRun. I understand HyperDriveStepRun's can be used to access all runs in the Hyperdrive step. Is this understanding accurate?

I'm also unsure how 1. create a new Experiment so that you can minimize such errors. should be setup or how it could help. Seek more clarification here.

Assuming 2. use HyperDriveRun directly. You can get the runid of the hyperdrive run from the Hyperdrive step run using get_children. I've attempted to write some code below replacing code in Step 3 in my pipeline, hyperdrive_save_best_model.py, is this what you're referring to?

# get run context
current_steprun = Run.get_context()
pipeline_runid = current_steprun.get_details()['properties']['azureml.pipelinerunid']
ws = current_steprun.experiment.workspace
pipeline_run = run.get_run(Experiment(ws, args.model_name),
                                             pipeline_runid
                                             ) 

# get the hyperdrive run id from the pipeline using the name of the step 'model_training_with_hyperdrive'
hyperdrive_run_id = pipeline_run.find_step_run('model_training_with_hyperdrive')[0]

# get pipeline experiment name
pipeline_experiment_name = args.model_name 

# get the drive run using the hyperdrive_run_id and pipeline experiment name
get_hyperdrive_run = HyperDriveRun(pipeline_experiment_name, hyperdrive_run_id)

# get the best model from the hyperdrive run
best_run = get_hyperdrive_run.get_best_run_by_primary_metric() 

[JackCaster]

Hi @julianaddison, any updates on this? I am also interested in retrieving the best model out of the HD step.

[LyaSolis]

+1