Evidence lower bound

Suppose we have a probabilistic model with observed variables X, latent variables Z and a joint distribution parametrized by θ. Typically, the goal is to maximize the likelihood function given by:

p(X|θ)=∑Zp(X,Z|θ).

We also suppose that direct optimization of p(X|θ) is difficult, but that optimization of p(X,Z|θ) is significantly easier. To proceed, we introduce a distribution q(Z) over the latent variables. We then have:

ln⁡p(X|θ)=ln⁡p(X,Z|θ)−ln⁡p(Z|X,θ)=ln⁡p(X,Z|θ)−ln⁡q(Z)−[ln⁡p(Z|X,θ)−ln⁡q(Z)]=ln⁡p(X,Z|θ)q(Z)−ln⁡p(Z|X,θ)q(Z)

Since the left-hand side does not depend on Z and q(Z) sums to one, we can take the expectation of both sides under q(Z):

ln⁡p(X|θ)=∑Zq(Z)ln⁡p(X|θ)=∑Zq(Z)ln⁡p(X,Z|θ)q(Z)⏟ELBO L(q,θ)−∑Zq(Z)ln⁡p(Z|X,θ)q(Z)⏟−KL(q∥p)

So

ln⁡p(X|θ)=L(q,θ)+KL(q∥p).

Since KL(q∥p)≥0, we have L(q,θ)≤ln⁡p(X|θ), i.e. L is a lower bound on the log evidence. The bound is tight when q(Z)=p(Z|X,θ), because the KL divergence is then zero.

The EM algorithm alternates directly setting q(Z)=p(Z|X,θcur) in the appropriate manner and optimizing L(q,θ) wrt θ while keeping q fixed to obtain θnext. After the E step, the lower bound takes the form:

L(q,θ)=∑Zp(Z|X,θcur)ln⁡p(X,Z|θ)−∑Zp(Z|X,θcur)ln⁡p(Z|X,θcur)=Q(θ,θcur)+const

Screenshot 2026-11-16 at 4.52.15 PM.png


Links

Sources