Energy-based models assign each input an unnormalized score, often through fθ(x) = exp(-Eθ(x)). Converting that score into a probability requires a partition function that is usually too expensive to calculate. A review paper by Luca Martino shows that several established workarounds share the same mathematical structure.

Martino connects noise-contrastive estimation (NCE), reverse logistic regression (RLR), Bennett’s acceptance ratio, multiple importance sampling (MIS), and bridge sampling. The paper identifies conditions under which their estimators are algebraically equivalent, derives additional weighting schemes from the shared framework, and releases MATLAB code for its numerical experiments.

Why one integral stalls training

An energy-based density has the form pθ(x) = fθ(x) / Zθ, where Zθ = ∫ fθ(x) dx. The model can evaluate fθ(x) at any chosen point, but computing Zθ requires integrating across the full input space. That integral becomes intractable for most high-dimensional models.

The unknown normalizer complicates several operations. Evaluating a normalized likelihood requires Zθ, while maximum-likelihood gradients require expectations under the model distribution, which developers usually approximate with Monte Carlo samples. Because Zθ changes with the model parameters, estimation errors can also propagate through training.

Several research communities address this problem through different formulations. NCE fits a classifier that distinguishes observed data from samples drawn from a known noise distribution. RLR uses logistic regression to estimate ratios of normalizing constants. Bridge sampling and Bennett’s acceptance ratio exploit overlap between a target and a reference distribution, while MIS combines weighted samples from one or more proposals.

Five names, one objective

Martino expresses the methods as minimizers of a common contrastive cost built from weighted samples associated with two distributions. One distribution contains the unknown normalizer; the other acts as a tractable proposal, noise source, or reference. Changing the weighting rule recovers NCE, RLR, MIS, and established bridge estimators.