cartpool-v2 need a fix on reward function probably?

[amugoodbad229]

I am learning RL on prototwin.

I think this line, reward -= self.reward_position_penalty() * reward * 0.5 should be reward -= self.reward_position_penalty() * 0.5, removing the reward .

Normal reward function for cartpole will look like:
reward = (position_weight * position_reward + angle_weight * angle_penalty + force_weight * force_penalty)

If that is not the case, can anyone explain why reward was added like this: reward -= self.reward_position_penalty() * reward * 0.5?

[kareemkermad]

The reward is initially determined from the angle of the pole from the upright position. The consequence of scaling the position penalty by the reward is that the cart gets penalized less for moving far away from the center, if the pole is already facing down.

def reward(self):
    '''
    Reward function
    '''
    dt = 0.005
    reward = 0
    reward += self.reward_angle()
    reward -= self.reward_position_penalty() * reward * 0.5
    reward -= self.reward_force_penalty() * 0.0025
    return reward * dt

This was done by design to ensure the cart has more leeway when swinging the pole up. The normal reward function you mentioned is what is used by cartpole-v1. If you compare the trained policies, you should see that v1 tries to keep the cart as close to the center as possible (at all times), whereas v2 initially moves the cart towards the end limit, but then moves it back to the center once the pole is balancing.

Obviously it does this so that it can swing the pole up faster, since it gets more reward this way and penalized less for doing so.