Exercise 2.1 without mujoco

[Bleyddyn]

Has anyone tried exercise 2.1 (Value Function Fitting in TRPO) with a non-mujoco environment?

I got it to run using the LunarLanderContinuous-v2 environment, however the outputs don't show much if any difference between the two value function tests (see the plot below, code is at https://github.com/Bleyddyn/spinningup/blob/master/my_solutions/exercise_2_1.ipynb).

Maybe I need to use a more complex environment that LunarLander? Unfortunately, I'm not sure what that would be. I plan on trying one or more of the roboschool envs next (as suggested in #35).

Edited to add:
Can someone who has a Mujoco license please post the plot to exercise 2.3? On Lunar Lander removing the action clipping step apparently has no effect and I'd like to see what it's supposed to look like. And the docs don't even list exercise 2.3 (https://spinningup.openai.com/en/latest/spinningup/exercises.html).

2nd Edit:
I still haven't gotten roboschool to work on my desktop or on Google Colab, so I can't test with any of the bullet based environments.

[abefetterman]

I've tried it with HopperBulletEnv-v0 from the pybullet_envs package, see the attached plot

You can merge my pr and then uncomment the 'pybullet_envs' line in user_config.py so it will work from the command line.

[Bleyddyn]

Thanks! I wonder why it doesn't work in LunarLander.