I am trying to understand the basics of empirical bayes. I found myself struggling a lot to understand this so I tried to create a toy example involving the estimation of the success probabilities for a coin.

1) Traditional Bayesian Analysis (Beta-Binomial Conjugacy):

Consider a sequence of $n$ coin tosses where we observe $X$ heads. The likelihood of observing $X$ heads given the true probability of heads $\theta$ is binomial: $$ P(X|\theta) = \binom{n}{X} \theta^X (1-\theta)^{n-X} $$

Chose a Beta prior distribution for the probability of heads $\theta$: $$ P(\theta) = \text{Beta}(\alpha, \beta) = \frac{\theta^{\alpha-1} (1-\theta)^{\beta-1}}{B(\alpha, \beta)} $$

Using Bayes' theorem, the posterior distribution is proportional to the likelihood times the prior: $$ P(\theta|X) \propto P(X|\theta) P(\theta) $$ Substituting the binomial likelihood and the Beta prior, we get: $$ P(\theta|X) \propto \theta^X (1-\theta)^{n-X} \cdot \theta^{\alpha-1} (1-\theta)^{\beta-1} $$ $$ P(\theta|X) \propto \theta^{X+\alpha-1} (1-\theta)^{n-X+\beta-1} $$ Thus, the posterior distribution is also a Beta distribution: $$ P(\theta|X) = \text{Beta}(X + \alpha, n - X + \beta) $$

2) Empirical Bayes:

It seems like in Empirical Bayes, the parameters of the priors are estimated from the data instead of being chosen before hand.

To me it makes more sense to use Method of Moments (instead of MLE) to estimate the parameters $\alpha$ and $\beta$ of the Beta prior.

The sample mean $\hat{\theta}$ and sample variance $s^2$ of the success probabilities can be used: $$ \hat{\theta} = \frac{\sum_{i} X_i}{\sum_{i} n_i} $$ $$ s^2 = \frac{\sum_{i} X_i (n_i - X_i)}{n_i^2 (n_i - 1)} $$

Using the method of moments, we equate the sample mean and variance to the mean and variance of the Beta distribution: $$ \hat{\theta} = \frac{\alpha}{\alpha + \beta} $$ $$ s^2 = \frac{\alpha \beta}{(\alpha + \beta)^2 (\alpha + \beta + 1)} $$

Solving these equations for $\alpha$ and $\beta$, we get: $$ \alpha = \hat{\theta} \left( \frac{\hat{\theta} (1 - \hat{\theta})}{s^2} - 1 \right) $$ $$ \beta = (1 - \hat{\theta}) \left( \frac{\hat{\theta} (1 - \hat{\theta})}{s^2} - 1 \right) $$

Now, we continue the Analysis as if it were regular Bayes:

The likelihood remains the same: $$ P(X_i|\theta_i) = \binom{n_i}{X_i} \theta_i^{X_i} (1 - \theta_i)^{n_i - X_i} $$

The prior is now based on the data: $$ P(\theta_i) = \text{Beta}(\hat{\alpha}, \hat{\beta}) $$

The posterior distribution is then: $$ P(\theta_i|X_i) \propto \theta_i^{X_i + \hat{\alpha} - 1} (1 - \theta_i)^{n_i - X_i + \hat{\beta} - 1} $$

$$ P(\theta_i|X_i) = \text{Beta}(X_i + \hat{\alpha}, n_i - X_i + \hat{\beta}) $$

Is this the correct idea? Empirical Bayes is the exact same thing as Traditional Bayesian.... but it uses the data to estimate the parameters of the prior?