If you want to use RMSProp for a tabular Q-learning algorithm, then you will have to first identify what the gradient is (the update rule - i.e. the gradient ascent - for Q-learning clearly suggests that the td_error is the gradient). Then, you just apply that to RMSProp by replacing the gradient g by the td_error in all RMSProp equations. So, what you are suggesting seems reasonable to me!

The only thing I can find to say is that you should be careful to keep a different moving average E[g^2] for each state-action pair (i.e. E[g^2] becomes a table, of the same size as Q)! (Note that the td_error is also state-action-dependent!) So, I would guess something like (with E[g^2] as M, for moving average):

M(s, a) <- beta * M(s, a) + (1-beta) * td_error(s, a)**2

Q(s,a) <- Q(s,a) + alpha_0 * td_error / np.sqrt(M(s, a)+epsilon).