PPG can update the value function in the auxiliary phase because it uses a technique called "reparameterization" which allows for the gradient of the value function to be computed more efficiently. Additionally, PPG can take advantage of the structure of the problem to provide a better estimate of the value function.

I think you can train the value network on data from any policy, but it will have more variance. If you have access to more data, it could help to smooth out the value function.

One approach would be to update the value network after each training phase of the policy network. This would ensure that the value network is always fitted to the current policy. However, this might not be practical if the training phases are very long.

Another approach would be to fit the value network using a rolling window of data from the previous training phases. This would strike a balance between stability and accuracy.

"Wouldn't it hurt the stability of fitting value function?"$\rightarrow$ $\color{red}{\textbf{No}}$, it should not hurt the stability of fitting the value function. In fact, using data from different policies can actually help to improve the stability of the value function estimate. This is because of "reparameterization" which allows for the gradient of the value function to be computed more efficiently as discussed above.