Question About Stage 2 Training

[hijukk]

Thank you for sharing this great work!
In the paper, the hyper-policy is trained with a two-stage reward, and it is stated that “the hyper-policy is continually updated.” Based on this description, I understood that Stage 2 training continues from the parameters learned in Stage 1, i.e., as a form of fine-tuning.

However, in the released code, it appears that:

• Stage 1 and Stage 2 training are executed separately
• In the Stage 2 training code, I could not find where the hyper-policy checkpoint from Stage 1 is loaded or resumed
• As a result, Stage 2 seems to start again from the same initial parameters as Stage 1

Is it possible that I missed a checkpoint load/resume code?

[huamo555]

Thank you for sharing this great work! In the paper, the hyper-policy is trained with a two-stage reward, and it is stated that “the hyper-policy is continually updated.” Based on this description, I understood that Stage 2 training continues from the parameters learned in Stage 1, i.e., as a form of fine-tuning.

However, in the released code, it appears that:

• Stage 1 and Stage 2 training are executed separately
• In the Stage 2 training code, I could not find where the hyper-policy checkpoint from Stage 1 is loaded or resumed
• As a result, Stage 2 seems to start again from the same initial parameters as Stage 1

Is it possible that I missed a checkpoint load/resume code?

Change the parameter in to and run the command:goal_condshadow_hand_residual_grasp.yaml False !!

[hijukk]

Thank you very much for the quick response!
I still have a few questions for clarification:

• Is the intended training procedure to run Stage 1 first, and then Stage 2 afterward?
• If so, should Stage 2 training start from the checkpoint obtained at the end of Stage 1?
• In the current released code, I could not find where the Stage 1 checkpoint is loaded when running Stage 2. It seems that Stage 2 starts from the same initialization as Stage 1.

Could you please let me know if I am missing a checkpoint loading step, or if the two stages are intentionally trained without explicitly resuming from a saved checkpoint?

Thank you again for your help!

[huamo555]

Thank you very much for the quick response! I still have a few questions for clarification:

• Is the intended training procedure to run Stage 1 first, and then Stage 2 afterward?
• If so, should Stage 2 training start from the checkpoint obtained at the end of Stage 1?
• In the current released code, I could not find where the Stage 1 checkpoint is loaded when running Stage 2. It seems that Stage 2 starts from the same initialization as Stage 1.

Could you please let me know if I am missing a checkpoint loading step, or if the two stages are intentionally trained without explicitly resuming from a saved checkpoint?

Thank you again for your help!

Based on my reproduction, they are indeed trained separately. First, set goal_cond to true, run the program, then set goal_cond to false and run it again; this requires two runs. I haven't looked at the code closely regarding the checkpoints you mentioned, but the final result is similar to what the author provided.

[analyst-huang]

Thank you very much for the quick response! I still have a few questions for clarification:

• Is the intended training procedure to run Stage 1 first, and then Stage 2 afterward?
• If so, should Stage 2 training start from the checkpoint obtained at the end of Stage 1?
• In the current released code, I could not find where the Stage 1 checkpoint is loaded when running Stage 2. It seems that Stage 2 starts from the same initialization as Stage 1.

Could you please let me know if I am missing a checkpoint loading step, or if the two stages are intentionally trained without explicitly resuming from a saved checkpoint?

Thank you again for your help!

First, set goal_cond to true and run the program. Then set goal_cond to false and run it again.
Ensure that the Stage 1 checkpoint is loaded when running Stage 2.