use cumulative environment steps as step in tensorboard

[AlexanderKoch-Koch]

I think it's more useful if the step in tensorboard corresponds to the total number of environment steps. This makes it easier to compare algorithms with different replay ratios.

[codecov-commenter]

Codecov Report

Merging #189 into master will increase coverage by 0.00%.
The diff coverage is 16.66%.

@@           Coverage Diff           @@
##           master     #189   +/-   ##
=======================================
  Coverage   20.98%   20.98%           
=======================================
  Files         132      132           
  Lines        8593     8595    +2     
=======================================
+ Hits         1803     1804    +1     
- Misses       6790     6791    +1     

Flag	Coverage Δ
#unittests	20.98% <16.66%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
rlpyt/runners/async_rl.py	0.00% <0.00%> (ø)
rlpyt/runners/minibatch_rl.py	43.84% <20.00%> (+0.06%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3996fca...f38551c. Read the comment docs.

[astooke]

This one is a bit too major of a change, I think we'll keep it as RL iterations, but I do usually end up plotting the x-axis as the logged CumSteps, as you're hoping for. Is there another way to do this in tensorboard?

[AlexanderKoch-Koch]

No, I think you can have only one x Axis in Tensorboard. We could of course make this different x Axis type optional. And add something like this to the logger: logger.set_tensorboard_step_type(type='cum_env_steps')
Do you think this solution is better?