Question regarding the calculation of the actor gradient.

[Tomakko]

Hi,

why did you include the minus sign in the grad_ys argument of the bottom function?

self.parameters_gradients = tf.gradients(self.action_output,self.parameters,-self.q_gradient_input/BATCH_SIZE)

As far as i understand grad_ys weights the gradients of each of the actor outputs with the the corresponding value (coming from the critic in your case).

Thanks!

[Tomakko]

Note that your are dividing twice through the batch size when computing the update for the actor weights.
1 q_gradient_batch = self.critic_network.gradients(state_batch,action_batch_for_gradients)/BATCH_SIZE in ddpg.py

2 self.parameters_gradients = tf.gradients(self.action_output,self.parameters,-self.q_gradient_input/BATCH_SIZE) in actor.py

This might be the cause of your bad performance.

[GeremWD]

Hi,
Did you manage to improve the performances by correcting this ? Because even if i do not see any other error in the code, it does not work.

[Tomakko]

Still embedding this into a bigger project. I can probably give you feedback if its working or not in a week or so.
Could you elaborate what it not working out? Does the networks converge?
Btw: Note that in the original paper the critic has a learning rate of 0.001 while the author is using 0.0001.

[doomie]

The paper suggests using batch-norm ("We also report results with components of our algorithm (i.e. the target network or batch normalization) removed. In order to perform
well across all tasks, both of these additions are necessary. "). See Figure 2 in https://arxiv.org/pdf/1509.02971.pdf

[floodsung]

Have fixed bugs and added batch norm on the actor network!