Evidence lower bound

The evidence lower bound (ELBO, also variational lower bound or negative variational free energy) is a lower bound on the probability of observing some data under a model. In Variational Bayesian methods, the ELBO can be used as an optimization criterion for approximating the posterior distribution $P(\mathbf {Z} |\mathbf {X} )$ of some set of latent variables $\mathbf {Z}$ given some data $\mathbf {X}$ with a simpler (parametric) distribution $Q(\mathbf {Z} )$ . These methods handle cases where a distribution $Q$ over unobserved variables $\mathbf {Z}$ is optimized as an approximation to the true posterior $P(\mathbf {Z} |\mathbf {X} )$ , given observed data $\mathbf {X}$ . Then the evidence lower bound is defined as:[1]

$L(X)=H(Q)-H(Q;P(X,Z))=-\sum _{\mathbf {Z} }Q(\mathbf {Z} )\log Q(\mathbf {Z} )+\sum _{\mathbf {Z} }Q(\mathbf {Z} )\log P(\mathbf {Z} ,\mathbf {X} )$

where $H(Q;P(X,Z))$ is cross-entropy. Maximizing the evidence lower bound minimizes $D_{\mathrm {KL} }(Q\parallel P)$ , the Kullback–Leibler divergence, a measure of dissimilarity of $Q$ from the true posterior. The primary reason why this quantity is preferred for optimization is that it can be computed without access to the posterior, given a good choice of $Q$ .

For other measures of dissimilarity to be optimized to fit $Q$ see Divergence (statistics).[2]

Justification as a lower bound on the evidence

The name evidence lower bound is justified by analyzing a decomposition of the KL-divergence between the true posterior and $Q$ :[3]

${\begin{aligned}D_{\mathrm {KL} }(Q\parallel P(Z|X))&=\sum _{\mathbf {Z} }Q(\mathbf {Z} )\left[\log {\frac {Q(\mathbf {Z} )P(\mathbf {X} )}{P(\mathbf {Z} ,\mathbf {X} )}}\right]\\D_{\mathrm {KL} }(Q\parallel P)&=\sum _{\mathbf {Z} }Q(\mathbf {Z} )\left[\log {\frac {Q(\mathbf {Z} )}{P(\mathbf {Z} ,\mathbf {X} )}}+\log P(\mathbf {X} )\right]\\D_{\mathrm {KL} }(Q\parallel P)&=\sum _{\mathbf {Z} }Q(\mathbf {Z} )\log Q(\mathbf {Z} )-\sum _{\mathbf {Z} }Q(\mathbf {Z} )\log P(\mathbf {Z} ,\mathbf {X} )+\log P(\mathbf {X} )\\\implies \log P(\mathbf {X} )-D_{\mathrm {KL} }(Q\parallel P)&=\sum _{\mathbf {Z} }Q(\mathbf {Z} )\log P(\mathbf {Z} ,\mathbf {X} )-\sum _{\mathbf {Z} }Q(\mathbf {Z} )\log Q(\mathbf {Z} )=:L(X)\end{aligned}}$

As $D_{\mathrm {KL} }(Q\parallel P)\geq 0$ this equation shows that the evidence lower bound is indeed a lower bound on the log-evidence $\log P(\mathbf {X} )$ for the model considered. As $\log P(\mathbf {X} )$ does not depend on $Q$ this equation additionally shows that maximizing the evidence lower bound on the right minimizes $D_{\mathrm {KL} }(Q\parallel P)$ , as claimed above.

References

Yang, Xitong. "Understanding the Variational Lower Bound" (PDF). Institute for Advanced Computer Studies. University of Maryland. Retrieved 20 March 2018.
Minka, Thomas (2005), Divergence measures and message passing. (PDF)
Bishop, Christopher M. (2006), "10.1 Variational Inference" (PDF), Pattern Recognition and Machine Learning

This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.

[1] Yang, Xitong. "Understanding the Variational Lower Bound" (PDF). Institute for Advanced Computer Studies. University of Maryland. Retrieved 20 March 2018.

[2] Minka, Thomas (2005), Divergence measures and message passing. (PDF)

[3] Bishop, Christopher M. (2006), "10.1 Variational Inference" (PDF), Pattern Recognition and Machine Learning