Conjugate Gaussian Models

  77 mins read  

My notes on Kevin Murphy’s Conjugate Bayesian Analysis of the Gaussian Distribution. I used his tutorial as my introduction to probability, linear algebra, and coding, but it was challenging since the notes are intended for a different audience. Hopefully someone finds mine to be a useful supplement to those.

If you spot an error or have ideas about making something clearer, please get in touch with me!

\(\require{color}\) $$ \require{cancel} \newcommand{\bm}{\boldsymbol} \newcommand{\circleEquation}[1] {\bbox[5px,border:2px solid black]{#1}} \newcommand{\given}[1][] {\:#1\vert\:} \definecolor{group4}{RGB}{114, 0, 172} \definecolor{group6}{RGB}{ 45, 177, 93} \definecolor{group3}{RGB}{251, 0, 29} \definecolor{group2}{RGB}{ 18, 110, 213} \definecolor{group5}{RGB}{217, 86, 16} \definecolor{group1}{RGB}{203, 23, 206} \newcommand{\prior}[1]{ p_0({#1}) } \newcommand{\likelihood}[2]{ p_{\ell}({#1} \given {#2}) } \newcommand{\posterior}[2]{ p_n({#1} \given {#2}) } \newcommand{\marginalLikelihood}[1]{ p_m({#1}) } \newcommand{\posteriorPredictive}[2]{ p_p({#1} \given {#2}) } \newcommand{\univariateGaussianEQ}[3]{ \frac{1}{\sqrt{2 \pi {#3}}} \exp{ \left\{ \frac{-1}{2 {#3}} ({#1} - {#2})^2 \right\} } } \newcommand{\multivariateGaussianEQ}[3]{ \frac{1 }{(2 \pi)^{D / 2} \begin{vmatrix} \bm{#3} \end{vmatrix}^{1/2} } \exp{ \left\{ \frac{-1}{2} (\bm{#1} - \bm{#2})^{\top}) \ \bm{#3}^{-1} \ (\bm{#1} - \bm{#2}) \right\} } } \newcommand{\normalAbbreviation}[3]{ \mathcal{N} \left({#1} \ \middle\vert \ {#2}, \ {#3}\right) } $$

Preliminaries

Concepts

Joint Probability (assuming Statistical Independence)


After watching the same movie, Matt is 10% likely to rate the movie an 8 while Sarah is 20% likely. What is the probability that both Matt and Sarah rate it an 8? If you answered 2%, then you assumed statistical independence:

$$ \begin{align} p(\mu = 8, \sigma = 8) &= \prior{\mu = 8} \times \prior{\sigma = 8} \\ &= .1 \times .2 \\ &= .02. \end{align} $$

is called the joint probability. A little more formally, two random variables and are statistically independent if the following equality is always true, for all values they can take on:

$$ \circleEquation{ p({\color{group1}\mu}, {\color{group2}\sigma}) = \prior{\color{group1}\mu} \times \prior{\color{group2}\sigma} }. $$

If there is statistical dependence between and , then , which requires an alternative way to compute the joint probability.


Conditional Probability


Suppose Matt and Sarah are 10% and 20% likely to rate a movie an 8, respectively, but Matt is a copycat: If Sarah rates it an 8, Matt is 40% likely to rate the movie an 8.

Conditional distributions such as specify different distributions depending on the conditions:

  • is the (likelihood) probability that Matt rates the movie an 8 if Sarah does.
  • is the (likelihood) probability that Matt rates the movie an 8 if Sarah does not.

is the overall probability that Matt is likely to rate the movie an 8.

Joint Probability (regardless of Statistical Independence)


If we know the likelihood of Matt rating the movie an 8 when Sarah rates it an 8, and the probability of Sarah rating it an 8 in the first place, we can compute the joint probability correctly, whether or not Matt’s response depends on Sarah’s:

$$ \circleEquation{ p({\color{group1}\mu}, {\color{group2}\sigma}) = \likelihood{\color{group1}\mu}{\color{group2}\sigma} \ \prior{\color{group2}\sigma} }. $$

Example: If and , then .

It might help to think of probabilities as the fraction of total space they occupy: Below, the total amount of area shared between the red and gray ovals is equal to the area of the red times the percent of the red shared with the gray.

  1. Let’s say the rectangular figure below has an area of 1, representing all of the possible outcomes.
  2. If we know the red region has of the area of the rectangle, the area of the red region is .
  3. Further, if we know of the red space is shared with the gray, the total amount of space occupied by both is the total amount of red times . Plugging in, we have .

figure_1


Bayes’ Rule


Let’s say we meet one of Matt’s friends who tells us that Matt thought the movie was pretty good. To get a clearer idea of whether this the movie is worth watching, we ask this friend how Matt might rate it on a 0-10 scale.

Bayes’ Rule tells us how to use this friend’s guess to update our belief about how Matt might rate the movie. Essentially, it exploits the fact that joint probabilities can be computed in two different ways.

:

  • is our confidence that Matt would rate the movie an 8 when the mutual friend says Matt thought the movie was "pretty good".
  • is the probability that this friend guesses 8, given Matt would have rated it an 8.

:

  • is the overall probability this friend would guess 8, regardless of how Matt rates the movie.
  • is the likelihood that that Matt would rate the movie an 8 if the friend guesses 8.
    • This is the thing we need to calculate. It tells us how confident we can be that Matt would rate the movie an 8 after his friend told us he thinks Matt would rate it an 8.
  • To make things easier to track later on, let’s give these unique subscripts:
    • and
    • .

Bayes rule is derived by setting the two ways of computing the joint probability equal to each other:

$$ \posterior{\color{group2} \mu }{\color{group1} x } \ \marginalLikelihood{\color{group1} x} = p(x, \mu) = \likelihood{\color{group1} x }{\color{group2} \mu } \ \prior{\color{group2} \mu} \\ \posterior{\color{group2} \mu }{\color{group1} x } \ \marginalLikelihood{\color{group1}x} = \likelihood{\color{group1} x }{\color{group2} \mu } \ \prior{\color{group2} \mu} \\ \circleEquation{ \posterior{\color{group2} \mu }{\color{group1} x } = \frac{ \likelihood{\color{group1} x}{\color{group2} \mu} \ \prior{\color{group2} \mu} }{\marginalLikelihood{\color{group1} x} } }. $$
  • is called the likelihood probability: probability that given .
  • is called the prior probability: overall probability that .
  • is called the posterior probability: probability that given .
  • is called the marginal probability aka (the evidence): overall probability that .
Conditional Independence


After Matt tells three of our mutual friends he liked the movie, we ask these friends how Matt might rate the movie out of 10. We will only watch the movie if we feel confident that Matt would rate it >8.

This means the guesses depend on what Matt might have rated the movie: .

Assuming Matt told all of his friends the same thing, the guesses our mutual friends make are independent of one another, allowing us to multiply conditional probabilities:

  • If Matt would rate the movie an 8, each of his friends are 50% likely to report a 7.
  • Then, the probability of all of them reporting a 7, given that Matt would rate the movie an 8 is .

The idea is the same as multiplying overall probabilities under statistical independence: If the probabilities are statistically independent given some information, we can multiply the conditional probabilities to obtain the joint conditional probability .

Gaussian Distribution


figure_2

The main thing to take away here is what the distribution looks like and how its shape is controlled:

  1. The mean has the highest probability density,
  2. "flatness" or "width" is controlled by the (co)variance parameter, and
  3. the normalizing constant outside of the exponentiation ensures that the area under the Gaussian is always 1.
ConceptUnivariate (1-Dimensional)Multivariate (>1-Dimensional)
Mean
(Co)variance
Normalizing
Constant

Univariate Equation

$$ \normalAbbreviation{x}{\mu}{\sigma^2} = \frac{1}{\sqrt{2 \pi \sigma^2}} e^{ \left\{ \frac{- 1}{2 \sigma^2} (x - \mu)^2 \right\}} $$

Multivariate Equation

$$ \normalAbbreviation{\bm{x}}{\bm{\mu}}{\bm{\Sigma}} = \frac{1}{(2 \pi)^{D / 2} \begin{vmatrix} \bm{\Sigma} \end{vmatrix}^{1/2}} e^{ \left\{ \frac{-1}{2} (\bm{x} - \bm{\mu})^{\top}) \ \bm{\Sigma}^{-1} \ (\bm{x} - \bm{\mu}) \right\}}, $$

where is a symmetric and positive definite matrix. You definitely need the symmetry property to make the multivariate derivations go anywhere. You may get away with not knowing what "positive definite" means, depending on what you are trying to get out of the notes.


Notation

Regular vs boldface font
  • is a number such as 0.5, but is a vector of numbers such as

    $$ \begin{bmatrix} -.25 & .1 & .32 & 5 \end{bmatrix}. $$
  • is a number such as 2, but is a matrix of numbers such as

    $$ \begin{bmatrix} 1 & 2 & 3 \\ 2 & 5 & 6 \\ 3 & 6 & 9 \end{bmatrix}. $$
Probability vs. probability distribution
  • Moving forward, if I follow a word for a distribution with the word "probability", I am referring to an actual number you obtain by plugging into and computing the function.
    • Prior probability
    • Posterior probability
    • Likelihood probability
    • Marginal likelihood probability
    • Posterior predictive probability
  • If I do not follow distribution names with the word "probability", I am referring to the whole distributions or density functions:
    • "Prior" means "prior distribution".
    • "Posterior" means "posterior distribution".
    • "Likelihood" means "likelihood distribution".
    • "Marginal likelihood" means "marginal likelihood distribution".
    • "Posterior predictive" means "posterior predictive distribution".
  • I use instead of because the larger font makes it easier to see what is going on inside of the exponent.
represents the space of possible values can take on.
  • In terms of die rolls, , the space of possible outcomes for a single die roll.
  • Here, in the univariate case, , where and .
    • Hence, , where .
  • In the multivariate case, , , and , where is the dimensionality of the data, so


My notation is slightly different from what you will typically find in papers — here is a table for reference.

PriorPosteriorLikelihoodMarginal
Likelihood
Posterior
Predictive


Objectives

To keep things in perspective, it helps to have end goals when working through math and code details.

Here, we are going to try to estimate what Matt thinks of a new movie he watched based on what he tells his friends. Since we don’t know Matt and none of our mutual friends have seen the movie yet, we ask these mutual friends what Matt thought of the movie. They say Matt thinks it is "pretty good", but that is not convincing, so we ask them how Matt might have rated the movie on a 0-10 scale. Using these data, we try to update our belief about how Matt would rate the movie.

After this, we want to infer the underlying distribution of the data we collect. We assume it is Gaussian with a mean, capturing information about how much Matt truly enjoys the movie, and a variance parameter, capturing information about the variability or reliability of the guesses.


Approach

Step 1: Assume the true likelihood distribution is Gaussian, with parameters .
  • If we are working with one dimensional data, the parameters are a mean and a variance: .
  • If we are working with >1-dimensional data, the parameters are a mean vector and a covariance matrix: .
  • Of course, assuming Gaussianity is a strong assumption, but we are just doing exercises here.
Step 2: Assume observed data are conditionally independent given the true Gaussian :
$$ \begin{align} \likelihood{\color{group1} x_1, x_2 \dots, n }{\color{group2} \Theta } &= \likelihood{\color{group1} x_1}{\color{group2} \Theta} \ \times \ \likelihood{\color{group1} x_2}{\color{group2} \Theta} \ \times \ \dots \ \times \ \likelihood{\color{group1} x_n}{\color{group2} \Theta} \\ &= \prod\limits_i^n \likelihood{ \color{group1} x_i }{\color{group2} \Theta } \\ &= \prod\limits_i^n {\color{red} \mathcal{N}} ({\color{group1} x_i} \given {\color{group2} \Theta}). \end{align} $$
Step 3: Put prior(s) on parameters you do not know (i.e. specify your certainty about their potential values).
  • For some of the models, either the mean or the covariance is assumed to be known (and therefore constant), but for others, both are assumed to be unknown.
$$ \likelihood{x_1, x_2, \dots, x_n}{\color{group2} \Theta} \times {\color{group2} \prior{\Theta}} $$
Step 4: Obtain the posterior using Bayes’ rule (i.e. update your certainty about the potential values of .
  • Just divide both sides by the marginal likelihood of the data.
$$ \posterior{ \color{group2} \Theta }{\color{group1} x_1, x_2, \dots, x_n } = \frac{ \likelihood{ \color{group1} x_1, x_2, \dots, x_n }{\color{group2} \Theta } \ \prior{\color{group2} \Theta } }{\marginalLikelihood{\color{group1} x_1, x_2, \dots, x_n} } $$
  • The marginal likelihood is obtained through marginalization of model parameters , which is just a fancy version of weighted summing of the likelihood probabilitites. It is analogous to computing a final grade for a course.
$$ \marginalLikelihood{\color{group1} x_1, x_2, \dots, x_n } = \int_{\mathfrak{D}({\color{group2} \Theta})} \likelihood{\color{group1} x_1, x_2, \dots, x_n }{\color{group2} \Theta } \ \prior{\color{group2} \Theta} d{\color{group2} \Theta} $$
  • For our purposes, solving this integral only requires a few algebra tricks.
Step 5: Using the likelihood and posterior, derive the posterior predictive—the distribution of interest.
  • Multiply the posterior and likelihood, and then integrate (i.e. marginalize) unknown parameters away.
    • Again, this is just a fancy version of weighted summing of the likelihood probabilities.
    • Recall that it is analogous to computing a final grade for a course.
  • The result is a formula for predicting new data points :
$$ \posteriorPredictive{ \color{group4} x_* }{\color{group1} x_1, x_2, \dots, x_n } = \int_{\mathfrak{D}({\color{group2} \Theta})} \likelihood{\color{group4} x_*}{\color{group2} \Theta} \ \posterior{\color{group2} \Theta }{\color{group1} x_1, x_2, \dots, x_n } d{\color{group2} \Theta}. $$
  • If we have enough data, the mean of this distribution should be close to or the same as the true mean (i.e. how Matt would have actually rated the move on a 0-10 scale).


Conjugate Gaussian Models

Model naming is based on the priors on the likelihood Gaussian. E.g. the model puts a Gaussian prior on the of the likelihood and assumes the of the likelihood is already known.

In case you are curious, one case in which we would know the true covariance of a distribution is after preprocessing, e.g. after “sphericalizing” or standardizing data to have a variance of 1 or an identity covariance matrix.

1D: Normal-Known

Likelihood of the Training Data
$$ \begin{align} p_{\color{group3} \ell} (x_1, x_2, \dots, x_n \given {\color{group6} \Theta}) &= \prod\limits_i^n \normalAbbreviation{x_i }{\color{group6} \mu_{\color{group3} \ell} }{\color{group6} \sigma_{\color{group3} \ell}^2 } \tag{1} \label{normalKnownLikelihood1}\\ &= \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} {\color{group1} \sum\limits_i^n (x_i - \mu_{\ell})^2 } \right\} } \tag{2} \label{normalKnownLikelihood2} \\ &= \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} {\color{group1} \sum\limits_i^n \left[ (x_i - \bar{x}) - (\mu_{\ell} - \bar{x})) \right]^2 } \right\}} \tag{3} \label{normalKnownLikelihood3} \\ &= \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{\left\{ \frac{-1}{2 \sigma_{\ell}^2} \left[ {\color{group2} ns^2} {\color{group5} + n(\bar{x} - \mu_{\ell})^2 } \right] \right\} } \tag{4} \label{normalKnownLikelihood4} \\ &= \circleEquation{ \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n {\color{group2} \exp{ \left\{ \frac{-ns^2}{2\sigma_{\ell}^2} \right\} } } {\color{group5} \left( \sqrt{\frac{2 \pi \sigma_{\ell}^2}{n}} \right) \normalAbbreviation{\bar{x} }{\mu_{\ell} }{\frac{\sigma_{\ell}^2}{n} } } } \tag{5} \label{normalKnownLikelihood5} \\ &\propto \normalAbbreviation{\bar{x}}{\mu_{\ell}}{\frac{\sigma_{\ell}^2}{n}} \tag{6} \label{normalKnownLikelihood6} \\ \end{align} $$
Equation 1: Assume the data is conditionally independent under a Gaussian.

From conditional independence, the (joint) likelihood of data is equal to the product of the individual likelihoods:

$$ \likelihood{ {\color{group1} x_1}, {\color{group1} x_2}, \dots, {\color{group1} x_n} }{\color{group2} \Theta } = \likelihood{\color{group1} x_1}{\color{group2} \Theta} \times \likelihood{\color{group1} x_2}{\color{group2} \Theta} \times \dots \times \likelihood{\color{group1} x_n}{\color{group2} \Theta}. $$

Since we assume the individual likelihoods are Gaussian, :

$$ {\color{group2} p_{\ell}}(x_i \given \Theta) \times {\color{group2} p_{\ell}}(x_2 \given \Theta) \times \dots \times {\color{group2} p_{\ell}}(x_n \given \Theta) \\ = {\color{group2} \mathcal{N}} (x_1 \given \mu_{\color{group2} \ell}, \sigma_{\color{group2} \ell}^2) \times {\color{group2} \mathcal{N}} (x_2 \given \mu_{\color{group2} \ell}, \sigma_{\color{group2} \ell}^2) \times \dots \times {\color{group2} \mathcal{N}} (x_n \given \mu_{\color{group2} \ell}, \sigma_{\color{group2} \ell}^2). $$

Rewrite the product using notation:

$$ \normalAbbreviation{\color{group5} x_1 }{\mu_{\ell} }{\sigma_{\ell}^2 } {\color{group5} \times} \normalAbbreviation{\color{group5} x_2 }{\mu_{\ell} }{\sigma_{\ell}^2 } {\color{group5} \times} \dots {\color{group5} \times} \normalAbbreviation{\color{group5} x_n }{\mu_{\ell} }{\sigma_{\ell}^2 } \\ = {\color{group5} \prod\limits_i^n} \normalAbbreviation{\color{group5} x_i}{\mu_{\ell}}{\sigma_{\ell}^2}. $$
Equation 2: Simplify constants and exponents in the product.

Write out the product by plugging into the Gaussian expression.

$$ \prod\limits_i^n \normalAbbreviation{\color{group1} x_i }{\color{group2} \mu_{\ell} }{\color{group2} \sigma_{\ell}^2 } = \univariateGaussianEQ{ \color{group1} x_1 }{\color{group2} \mu_{\ell} }{\color{group2} \sigma_{\ell}^2 } \times \univariateGaussianEQ{ \color{group1} x_2 }{\color{group2} \mu_{\ell} }{\color{group2} \sigma_{\ell}^2 } \times \dots \times \univariateGaussianEQ{ \color{group1} x_n }{\color{group2} \mu_{\ell} }{\color{group2} \sigma_{\ell}^2 } $$

Bring the normalizing constants to the front.

$$ {\color{group5} \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} } \exp{ \left\{ \frac{-1 }{2\sigma_{\ell}^2 } (x_1 - \mu_{\ell})^2 \right\} } \times {\color{group5} \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} } \exp{ \left\{ \frac{-1 }{2\sigma_{\ell}^2 } (x_2 - \mu_{\ell})^2 \right\} } \times \dots \times {\color{group5} \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} } \exp{ \left\{ \frac{-1 }{2\sigma_{\ell}^2 } (x_n - \mu_{\ell})^2 \right\} } \\ = {\color{group5} \left( \frac{1 }{\sqrt{2 \pi \sigma_{\ell}^2} } \right)^n } \exp{ \left\{ \frac{-1 }{2\sigma_{\ell}^2 } (x_1 - \mu_{\ell})^2 \right\} } \times \dots \times \exp{ \left\{ \frac{-1 }{2\sigma_{\ell}^2 } (x_n - \mu_{\ell})^2 \right\} } $$

Sum exponents. Recall that .

$$ \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1 }{2\sigma_{\ell}^2 } (x_1 - \mu_{\ell})^2 \right\} } {\color{group5} \times} \exp{ \left\{ \frac{-1 }{2\sigma_{\ell}^2 } (x_2 - \mu_{\ell})^2 \right\} } {\color{group5} \times} \dots {\color{group5} \times} \exp{ \left\{ \frac{-1 }{2\sigma_{\ell}^2 } (x_n - \mu_{\ell})^2 \right\} } \\ = \left( \frac{1 }{\sqrt{2 \pi \sigma_{\ell}^2} } \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} (x_1 - \mu_{\ell})^2 {\color{group5} +} \frac{-1}{2 \sigma_{\ell}^2} (x_2 - \mu_{\ell})^2 {\color{group5} +} \dots {\color{group5} +} \frac{-1}{2 \sigma_{\ell}^2} (x_n - \mu_{\ell})^2 \right\} } $$

Factor out within the exponent.

$$ \left( \frac{1 }{\sqrt{2 \pi \sigma_{\ell}^2} } \right)^n \exp{\left\{ {\color{group5} \frac{-1}{2 \sigma_{\ell}^2}} (x_1 - \mu_{\ell})^2 + {\color{group5} \frac{-1}{2 \sigma_{\ell}^2}} (x_2 - \mu_{\ell})^2 + \dots + {\color{group5} \frac{-1}{2 \sigma_{\ell}^2}} (x_n - \mu_{\ell})^2 \right\} } \\ = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ {\color{group5} \frac{-1}{2 \sigma_{\ell}^2}} \left[ (x_1 - \mu_{\ell})^2 + (x_2 - \mu_{\ell})^2 + \dots + (x_n - \mu_{\ell})^2 \right] \right\} } $$

Represent the sum using notation.

$$ \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \left[ (x_{\color{group5} 1} - \mu_{\ell})^2 {\color{group5} +} (x_{\color{group5} 2} - \mu_{\ell})^2 {\color{group5} +} \dots {\color{group5} +} (x_{\color{group5} n} - \mu_{\ell})^2 \right] \right\} } \\ = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} {\color{group5} \sum\limits_{i}^n} (x_{\color{group5} i} - \mu_{\ell})^2 \right\} } $$
Equation 3: Introduce the empirical mean .

Introduce 0:

$$ \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n {\color{group5}(x_i - \mu_{\ell})}^2 \right\} } = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n {\color{group5} (x_i + 0 - \mu_{\ell})}^2 \right\} }. $$

Since , where , we can introduce the empirical mean the following way:

$$ \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n (x_i + {\color{group5} 0} - \mu_{\ell})^2 \right\} } \\ = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n \left( x_i - {\color{group5}\bar{x} + \bar{x}} - \mu_{\ell} \right)^2 \right\} } \\ = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n \left[ (x_i - \bar{x}) + (\bar{x} - \mu_{\ell}) \right]^2 \right\} }. $$

Since ,

$$ \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n \left[ (x_i - \bar{x}) {\color{group5} + (\bar{x} - \mu_{\ell})} \right]^2 \right\} } \\ = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n \left[ (x_i - \bar{x}) {\color{group5} - (\mu_{\ell} - \bar{x})} \right]^2 \right\} }. $$
Equation 4: Introduce the empirical variance .

Expand the quadratic expression inside of the summation. Recall that . Here, , :

$$ \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n \left[ {\color{group1} (x_i - \bar{x}) } - {\color{group2} (\mu_{\ell} - \bar{x}) } \right]^2 \right\} } \\ = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n \left[ {\color{group1} (x_i - \bar{x}) }^2 - 2 { \color{group1} (x_i - \bar{x}) } { \color{group2} (\mu_{\ell} - \bar{x}) } + {\color{group2} (\mu_{\ell} - \bar{x})}^2 \right] \right\} } $$

Since , we obtain

$$ \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} {\color{group5} \sum\limits_i^n} \left[ (x_i - \bar{x})^2 - 2 (x_i - \bar{x}) (\mu_{\ell} - \bar{x}) + (\mu_{\ell} - \bar{x})^2 \right] \right\} } \\ = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \left[ {\color{group5} \sum\limits_i^n} (x_i - \bar{x})^2 - {\color{group5} \sum\limits_i^n} 2(x_i - \bar{x}) (\mu_{\ell} - \bar{x}) + {\color{group5} \sum\limits_i^n} (\mu_{\ell} - \bar{x})^2 \right] \right\} }. $$

If we define the empirical variance as , multiplying both sides by yields . We can substitute this for the first term in the exponent:

$$ \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \left[ {\color{group1} \sum\limits_i^n (x_i - \bar{x})^2} - \sum\limits_i^n 2(x_i - \bar{x})(\mu_{\ell} - \bar{x}) + \sum\limits_i^n (\mu_{\ell} - \bar{x})^2 \right] \right\} } \\ = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \left[ {\color{group1} ns^2} - \sum\limits_i^n 2(x_i - \bar{x})(\mu_{\ell} - \bar{x}) + \sum\limits_i^n (\mu_{\ell} - \bar{x})^2 \right] \right\} }. $$

Factor out from the second term in the exponent.

$$ \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \left[ ns^2 - \sum\limits_i^n {\color{group2} 2} (x_i - \bar{x}) {\color{group2} (\mu_{\ell} - \bar{x})} + \sum\limits_i^n (\mu_{\ell} - \bar{x})^2 \right] \right\} } \\ = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \left[ ns^2 - {\color{group2} 2 (\mu_{\ell} - \bar{x})} \sum\limits_i^n (x_i - \bar{x}) + \sum\limits_i^n (\mu_{\ell} - \bar{x})^2 \right] \right\} }. $$

Since the second term in the exponent becomes zero.

$$ \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \left[ ns^2 - {\color{group2} 2(\mu_{\ell} - \bar{x})} {\color{group5} \sum\limits_i^n (x_i - \bar{x})} + \sum\limits_i^n (\mu_{\ell} - \bar{x})^2 \right] \right\} } \\ = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \left[ ns^2 - {\color{group2}2(\mu_{\ell} - \bar{x})} {\color{group5} \times 0} + \sum\limits_i^n (\mu_{\ell} - \bar{x})^2 \right] \right\} } \\ = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \left[ ns^2 + {\color{group5}0} + \sum\limits_i^n (\mu_{\ell} - \bar{x})^2 \right] \right\} }. $$

Finally, since ,

$$ \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \left[ ns^2 + {\color{group2} \sum\limits_i^n (\mu_{\ell} - \bar{x})^2 } \right] \right\} } \\ = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \left[ ns^2 + {\color{group2} n(\mu_{\ell} - \bar{x})^2} \right] \right\} }. $$
Equation 5: Rewrite the empirical mean as a Gaussian distributed random variable.

Distribute the constant containing the likelihood variance :

$$ \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ {\color{group5} \frac{-1}{2 \sigma_{\ell}^2}} \left[ ns^2 + n(\mu_{\ell} - \bar{x})^2 \right] \right\} } \\ = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{ns^2}{\color{group5} -2 \sigma_{\ell}^2} + \frac{n}{\color{group5} -2 \sigma_{\ell}^2} (\mu_{\ell} - \bar{x})^2 \right\} } $$

Since ,

$$ \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{ns^2}{-2 \sigma_{\ell}^2} {\color{group5} +} \frac{n}{-2 \sigma_{\ell}^2} (\mu_{\ell} - \bar{x})^2 \right\} } \\ = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{ns^2}{-2 \sigma_{\ell}^2} \right\} } {\color{group5} \times} \exp{ \left\{ \frac{n}{-2 \sigma_{\ell}^2} (\mu_{\ell} - \bar{x})^2 \right\} }. $$

Since , we can rewrite the third expression as a Gaussian with variance by introducing

$$ \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{ns^2}{-2 \sigma_{\ell}^2} \right\} } \times \exp{ \left\{ \frac{n}{-2 \sigma_{\ell}^2} (\mu_{\ell} - \bar{x})^2 \right\} } \\ = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{ns^2}{-2 \sigma_{\ell}^2} \right\} } \times {\color{group5} 1} \times \exp{ \left\{ \frac{n}{-2 \sigma_{\ell}^2} (\mu_{\ell} - \bar{x})^2 \right\} } \\ = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{ns^2}{-2 \sigma_{\ell}^2} \right\} } \times {\color{group5} \frac{\sqrt{2 \pi \frac{\sigma_{\ell}^2}{n}} }{\sqrt{2 \pi \frac{\sigma_{\ell}^2}{n}} } } \times \exp{ \left\{ \frac{\color{group4} n }{-2 {\color{group4}\sigma_{\ell}^2} } (\mu_{\ell} - \bar{x})^2 \right\} } \\ = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{ns^2}{-2 \sigma_{\ell}^2} \right\} } \times {\color{group5} \left( \sqrt{\frac{2 \pi \sigma_{\ell}^2}{n}} \right) \left( \frac{1 }{\sqrt{\frac{2 \pi \sigma_{\ell}^2}{n}} } \right) } \times \exp{ \left\{ \frac{1 }{-2 {\color{group4} \frac{\sigma_{\ell}^2}{n}} } (\mu_{\ell} - \bar{x})^2 \right\} } \\ $$

Abbreviating the Gaussian expression,

$$ \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{ns^2}{-2 \sigma_{\ell}^2} \right\} } \times \left( \sqrt{\frac{2 \pi \sigma_{\ell}^2}{n}} \right) {\color{group5} \left( \frac{1 }{\sqrt{\frac{2 \pi \sigma_{\ell}^2}{n}} } \right) \times \exp{ \left\{ \frac{1}{-2 \frac{\sigma_{\ell}^2}{n}} (\mu_{\ell} - \bar{x})^2 \right\} } } \\ = \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-ns^2}{2 \sigma_{\ell}^2} \right\} } \times \sqrt{\frac{2 \pi \sigma_{\ell}^2}{n}} \times {\color{group5} \normalAbbreviation{\bar{x} }{\mu_{\ell} }{\frac{\sigma_{\ell}^2}{n} } }. $$
Equation 6: Note the proportionality of the Gaussian to the joint likelihood.

Recall that there is only one variable in this model: . Everything else, including , , is known, i.e. constant, which means is a constant, making the fourth factor directly proportional to the joint likelihood.

$$ \begin{align} {\color{group5} \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{\left\{ \frac{-ns^2}{2 \sigma_{\ell}^2} \right\} } \left( \sqrt{\frac{n}{2 \pi \sigma_{\ell}^2}} \right) } \normalAbbreviation{\bar{x}}{\mu_{\ell}}{\frac{\sigma_{\ell}^2}{n}} &{\color{group5} = c} \times \normalAbbreviation{\bar{x} }{\mu_{\ell} }{\frac{\sigma_{\ell}^2}{n} } \\ &{\color{group5} \propto \ } \normalAbbreviation{\bar{x}}{\mu_{\ell}}{\frac{\sigma_{\ell}^2}{n}} \end{align} $$
Prior on Likelihood Parameters
$$ \begin{align} \prior{\Theta} &= \prior{\mu_{\ell}, \ \sigma_{\ell}^2} \tag{1} \label{normalKnownPrior1} \\ &= \prior{\mu_{\ell} \given \sigma_{\ell}^2} \times \prior{\sigma_{\ell}^2} \tag{2} \label{normalKnownPrior2}\\ &= \prior{\mu_{\ell} \given \sigma_{\ell}^2} \tag{3} \label{normalKnownPrior3} \\ &= \circleEquation{ \normalAbbreviation{\mu_{\ell} }{\mu_0 }{\color{group1} \frac{\sigma_{\ell}^2}{\kappa_0} } } \tag{4} \label{normalKnownPrior4} \\ &= \normalAbbreviation{\mu_{\ell} }{\mu_0 }{\color{group1} \sigma_0^2 }. \tag{5} \label{normalKnownPrior5} \\ \end{align} $$
Equation 1: Plug in for .

Recall that I defined it the following way in Step 1 of the Approach.

$$ \prior{\color{group1} \Theta} = \prior{\color{group1} \mu_{\ell}, \ \sigma_{\color{group1} \ell}^2} $$
Equation 2: Rewrite the joint distribution as a product of the conditional and corresponding marginal.
$$ \prior{ {\color{group1} \mu_{\ell}}, \ {\color{group2} \sigma_{\ell}^2} } = \prior{ {\color{group1}\mu_{\ell}} \given {\color{group2} \sigma_{\ell}^2} } \times \prior{\color{group2} \sigma_{\ell}^2} $$
Equation 3: is given, meaning we know it with 100% certainty, so .
$$ \begin{align} \prior{\mu_{\ell} \given \sigma_{\ell}^2} {\color{group1} \times \prior{\sigma_{\ell}^2} } &= \prior{ \mu_{\ell} \given {\color{group1} \sigma_{\ell}^2} } {\color{group1} \times 1} \\ &= \prior{\mu_{\ell} \given \sigma_{\ell}^2}. \end{align} $$
Equation 4: Magic!

What we want to do now is specify a distribution of our choice on . However, notice that appears as a parameter in the prior. This is weird because we typically want flexibility in specifying a prior variance . For example, may want to use because we are very uncertain what the likelihood mean is.

One trick to recover the flexibility in choosing the prior variance is to let :

$$ \prior{\mu_{\ell} \given \sigma_{\ell}^2} = \normalAbbreviation{\mu_{\ell} }{\mu_0 }{\color{group2} \frac{\sigma_{\ell}^2}{\kappa_0} }. $$

Now, the prior depends on , since we defined it that way. However, it is also as "flexible" as if it does not: if we want a large variance in the prior, we pick a small and if we want a small variance, we pick a large That is, for practical purposes, we can still choose any by plugging in the known variance , the chosen prior covariance , and solving for the appropriate using the formula .

Equation 5: The likelihood mean is assumed to be sampled from a Gaussian prior with parameters and .

To make derivations clearer in the following sections, we use instead of from the formula
:

$$ \normalAbbreviation{\mu_{\ell} }{\mu_0 }{\color{group2} \frac{\sigma_{\ell}^2}{\kappa_0} } = \normalAbbreviation{\mu_{\ell}}{\mu_0}{\color{group2} \sigma_0^2}. $$
Marginal Likelihood of the Training Data
$$ \begin{align} \marginalLikelihood{x_1, x_2, \dots, x_n} &= \int_{-\infty}^{+\infty} \left[ \prod\limits_i^n \normalAbbreviation{x_i}{\mu_{\ell}}{\sigma_{\ell}^2} \right] \times \normalAbbreviation{\mu_{\ell}}{\mu_0}{\sigma_0^2} d\mu_{\ell} \tag{1} \label{normalKnownMarginalLikelihood1} \\ &= \int_{-\infty}^{+\infty} C_1 \times \exp{\left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n (x_i - \mu_{\ell})^2 - \frac{1}{2 \sigma_0^2} (\mu_{\ell} - \mu_0)^2 \right\}} d\mu_{\ell} \tag{2} \label{normalKnownMarginalLikelihood2} \\ &= \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{\color{group6} \mu_{\ell}^2}{2} {\color{group1} \left( \frac{n}{\sigma_{\ell}^2} + \frac{1}{\sigma_0^2} \right) } + {\color{group6} \mu_{\ell}} \left( \frac{n \bar{x}}{\sigma_{\ell}^2} + \frac{\mu_0}{\sigma_0^2} \right) \right\} } \times C_2 d\mu_{\ell} \tag{3} \label{normalKnownMarginalLikelihood3} \\ &= \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{\color{group6} \mu_{\ell}^2}{2 \color{group1} \sigma_n^2} + \frac{\color{group6} \mu_{\ell}}{\color{group1} \sigma_n^2} \left( {\color{group2} \frac{n \bar{x} \sigma_0^2 + \mu_0 \sigma_{\ell}^2 }{n \sigma_0^2 + \sigma_{\ell}^2 } } \right) \right\} } \times C_2 d\mu_{\ell} \tag{4} \label{normalKnownMarginalLikelihood4} \\ &= \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{1}{2 {\color{group1} \sigma_n^2}} \left[ {\color{group6} \mu_{\ell}^2} - 2 {\color{group6} \mu_{\ell}} {\color{group2} \mu_n} + {\color{group2} \mu_n}^2 \right] \right\} } \times C_3 d\mu_{\ell} \tag{5} \label{normalKnownMarginalLikelihood5} \\ &= C_1 \times C_3 \times \sqrt{2 \pi \sigma_n^2} \tag{6} \label{normalKnownMarginalLikelihood6} \\ &= \circleEquation{ \sqrt{ \frac{\sigma_{\ell}^2 }{\sigma_n^2 (2 \pi \sigma_{\ell}^2)^n } } \times \exp{ \left\{ \frac{ \left( \frac{n \bar{x} \sigma_0}{\sigma_{\ell}} + \frac{\mu_0 \sigma_{\ell}}{\sigma_0} \right)^2 }{2 \sigma_n^2 } \right\} } \times \exp{ \left\{ - \frac{\sum\limits_i^n x_i^2 }{2 \sigma_{\ell}^2 } - \frac{\mu_0^2 }{2 \sigma_0^2 } \right\} } } \tag{7} \label{normalKnownMarginalLikelihood7} \\ \end{align} $$
Equation 1: Plug in the joint likelihood of the training data and the prior.

As I stated in the Step 4 of the approach, the marginal likelihood is defined as follows:

$$ \marginalLikelihood{x_1, x_2, \dots, x_n} = \int_{\color{group2} \mathfrak{D}(\Theta)} \likelihood{x_1, x_2, \dots, x_n }{\color{group2} \Theta } \times \prior{\color{group2} \Theta} d{\color{group2} \Theta}. $$

Plug in the joint likelihood obtained in Equation 1 of the section deriving the Likelihood of the Training Data:

$$ \int_{\mathfrak{D}(\Theta)} {\color{group5} \likelihood{x_1, x_2, \dots, x_n }{\Theta } } \times \prior{\Theta} d\Theta \\ = \int_{\mathfrak{D}(\Theta)} {\color{group5} \left[ \prod\limits_i^n \normalAbbreviation{x_i}{\mu_{\ell}}{\sigma_{\ell}^2} \right] } \times \prior{\Theta} d\Theta. \\ $$

Plug in the result from Equation 5 of the section deriving the prior.

$$ \int_{\mathfrak{D}(\Theta)} \left[ \prod\limits_i^n \normalAbbreviation{x_i}{\mu_{\ell}}{\sigma_{\ell}^2} \right] \times {\color{group5} \prior{\Theta}} d\Theta \\ = \int_{\mathfrak{D}(\Theta)} \left[ \prod\limits_i^n \normalAbbreviation{x_i}{\mu_{\ell}}{\sigma_{\ell}^2} \right] \times {\color{group5} \normalAbbreviation{\mu_{\ell}}{\mu_0}{\sigma_0^2} } d\Theta \\ $$

For the Normal-Known model, the likelihood variance is constant (known with probability 1), meaning there is no prior on the likelihood variance. The only unknown variable is the likelihood mean ; hence,

$$ \int_{\color{group5} \mathfrak{D}(\Theta)} \left[ \prod\limits_i^n \normalAbbreviation{x_i}{\mu_{\ell}}{\sigma_{\ell}^2} \right] \times \normalAbbreviation{\mu_{\ell}}{\mu_0}{\sigma_0^2} {\color{group5} d\Theta} \\ = \int_{\color{group5} -\infty}^{\color{group5} +\infty} \left[ \prod\limits_i^n \normalAbbreviation{x_i}{\mu_{\ell}}{\sigma_{\ell}^2} \right] \times \normalAbbreviation{\mu_{\ell}}{\mu_0}{\sigma_0^2} {\color{group5} d\mu_{\ell}}. \\ $$
Equation 2: Rewrite the product as a sum of exponents.

Plugging in the result in Equation 2 of the section deriving the Likelihood of the Training Data, we can rewrite the product in the brackets as a sum of exponents:

$$ \int_{-\infty}^{+\infty} {\color{group5} \left[ \prod\limits_i^n \normalAbbreviation{x_i}{\mu_{\ell}}{\sigma_{\ell}^2} \right] } \times \normalAbbreviation{\mu_{\ell}}{\mu_0}{\sigma_0^2} d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} {\color{group5} \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n (x_i - \mu_{\ell})^2 \right\} } } \normalAbbreviation{\mu_{\ell}}{\mu_0}{\sigma_0^2} d\mu_{\ell}. $$

Plugging in the definition of the Gaussian, we have

$$ \int_{-\infty}^{+\infty} \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n (x_i - \mu_{\ell})^2 \right\} } {\color{group5} \normalAbbreviation{\mu_{\ell}}{\mu_0}{\sigma_0^2} } d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n (x_i - \mu_{\ell})^2 \right\} } \times {\color{group5} \univariateGaussianEQ{\mu_{\ell}}{\mu_0}{\sigma_0^2} } d\mu_{\ell}. $$

Let

$$ \int_{-\infty}^{+\infty} {\color{group5} \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n } \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n (x_i - \mu_{\ell})^2 \right\} } \times {\color{group5} \frac{1}{\sqrt{2 \pi \sigma_0^2}} } \exp{ \left\{ \frac{-1}{2 \sigma_0^2} (\mu_{\ell} - \mu_0)^2 \right\} } d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} {\color{group5} C_1 \times} \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n (x_i - \mu_{\ell})^2 \right\} } \times \exp{ \left\{ \frac{-1}{2 \sigma_0^2} (\mu_{\ell} - \mu_0)^2 \right\} } d\mu_{\ell}. $$

Rewriting the product of the two exponents as a sum of exponents, we have

$$ \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n (x_i - \mu_{\ell})^2 \right\} } {\color{group5} \times} \exp{ \left\{ \frac{-1}{2 \sigma_0^2} (\mu_{\ell} - \mu_0)^2 \right\} } d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n (x_i - \mu_{\ell})^2 {\color{group5} +} \frac{1}{- 2 \sigma_0^2} (\mu_{\ell} - \mu_0)^2 \right\} } d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ \frac{1}{- 2 \sigma_{\ell}^2} \sum\limits_i^n (x_i - \mu_{\ell})^2 {\color{group5} -} \frac{1}{2 \sigma_0^2} (\mu_{\ell} - \mu_0)^2 \right\} } d\mu_{\ell}. $$
Equation 3: Factor out , the only variable.

Expand both of the quadratic expressions in the exponent:

$$ \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n {\color{group5} (x_i - \mu_{\ell})^2} - \frac{1}{2 \sigma_0^2} {\color{group5} (\mu_{\ell} - \mu_0)^2} \right\} } d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \sum\limits_i^n {\color{group5} [x_i^2 - 2 x_i \mu_{\ell} + \mu_{\ell}^2]} - \frac{1}{2 \sigma_0^2} {\color{group5} [\mu_{\ell}^2 - 2 \mu_{\ell} \mu_0 + \mu_0^2]} \right\} } d\mu_{\ell}. $$

Distribute the summation:

$$ \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} {\color{group5} \sum\limits_i^n} [x_i^2 - 2 x_i \mu_{\ell} + \mu_{\ell}^2] - \frac{1}{2 \sigma_0^2} [\mu_{\ell}^2 - 2 \mu_{\ell} \mu_0 + \mu_0^2] \right\} } d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \left[ {\color{group5} \sum\limits_i^n} x_i^2 - {\color{group5} \sum\limits_i^n} 2 x_i \mu_{\ell} + {\color{group5} \sum\limits_i^n} \mu_{\ell}^2 \right] - \frac{1}{2 \sigma_0^2} [\mu_{\ell}^2 - 2 \mu_{\ell} \mu_0 + \mu_0^2] \right\} } d\mu_{\ell}. $$

Since and ,

$$ \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} \left[ \sum\limits_i^n x_i^2 - {\color{group5} \sum\limits_i^n 2 x_i} \mu_{\ell} + {\color{group5} \sum\limits_i^n} \mu_{\ell}^2 \right] - \frac{1}{2 \sigma_0^2} [\mu_{\ell}^2 - 2 \mu_{\ell} \mu_0 + \mu_0^2] \right\} } d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ \frac{-1}{\color{group1} 2 \sigma_{\ell}^2} \left[ \sum\limits_i^n x_i^2 - {\color{group5} 2n \bar{x}} \mu_{\ell} + {\color{group5} n} \mu_{\ell}^2 \right] - \frac{1}{\color{group2} 2 \sigma_0^2} [\mu_{\ell}^2 - 2 \mu_{\ell} \mu_0 + \mu_0^2] \right\} } d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ \frac{\sum\limits_i^n x_i^2}{\color{group1} - 2 \sigma_{\ell}^2} + \frac{\cancel{2} n \bar{x} \mu_{\ell} }{\color{group1} \cancel{2} \sigma_{\ell}^2 } - \frac{n \mu_{\ell}^2}{\color{group1} 2 \sigma_{\ell}^2} - \frac{\mu_{\ell}^2}{\color{group2} 2 \sigma_0^2} + \frac{\cancel{2} \mu_{\ell} \mu_0 }{\color{group2} \cancel{2} \sigma_0^2 } - \frac{\mu_0^2}{\color{group2} 2 \sigma_0^2} \right\} } d\mu_{\ell}. $$

We can begin to complete the square by grouping like terms and factoring:

$$ \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ \frac{\sum\limits_i^n x_i^2}{-2 \sigma_{\ell}^2} + {\color{group5} \frac{n \bar{x} {\color{group6} \mu_{\ell}} }{\sigma_{\ell}^2 } } - {\color{group1} \frac{n {\color{group2} \mu_{\ell}^2} }{2 \sigma_{\ell}^2 } } - {\color{group1} \frac{\color{group2} \mu_{\ell}^2 }{2 \sigma_0^2 } } + {\color{group5} \frac{\mu_0 \color{group6} \mu_{\ell} }{\sigma_0^2 } } - \frac{\mu_0^2}{2 \sigma_0^2} \right\} } d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - {\color{group2} \mu_{\ell}^2} {\color{group1} \left( \frac{n}{2 \sigma_{\ell}^2} + \frac{1}{2 \sigma_0^2} \right) } + {\color{group6} \mu_{\ell}} {\color{group5} \left( \frac{n \bar{x}}{\sigma_{\ell}^2} + \frac{\mu_0}{\sigma_0^2} \right) } - \frac{\sum\limits_i^n x_i^2}{2 \sigma_{\ell}^2} - \frac{\mu_0^2}{2 \sigma_0^2} \right\} } d\mu_{\ell}. $$

Letting and factoring from the first term,

$$ \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \mu_{\ell}^2 \left( \frac{n}{\color{group5} 2 \sigma_{\ell}^2} + \frac{1}{\color{group5} 2 \sigma_0^2} \right) + \mu_{\ell} \left( \frac{n \bar{x}}{\sigma_{\ell}^2} + \frac{\mu_0}{\sigma_0^2} \right) - \frac{\sum\limits_i^n x_i^2}{2 \sigma_{\ell}^2} - \frac{\mu_0^2}{2 \sigma_0^2} \right\} } d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{\mu_{\ell}^2}{\color{group5} 2} \left( \frac{n}{\color{group5} \sigma_{\ell}^2} + \frac{1}{\color{group5} \sigma_0^2} \right) + \mu_{\ell} \left( \frac{n \bar{x}}{\sigma_{\ell}^2} + \frac{\mu_0}{\sigma_0^2} \right) \right\} } \times {\color{group1} \exp{ \left\{ - \frac{\sum\limits_i^n x_i^2}{2 \sigma_{\ell}^2} - \frac{\mu_0^2}{2 \sigma_0^2} \right\} } } d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{\mu_{\ell}^2}{2} \left( \frac{n}{\sigma_{\ell}^2} + \frac{1}{\sigma_0^2} \right) + \mu_{\ell} \left( \frac{n \bar{x}}{\sigma_{\ell}^2} + \frac{\mu_0}{\sigma_0^2} \right) \right\} } \times {\color{group1} C_2} d\mu_{\ell}. $$
Equation 4: Factor out .

Let :

$$ \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{\mu_{\ell}^2}{2} {\color{group5} \left( \frac{n}{\sigma_{\ell}^2} + \frac{1}{\sigma_0^2} \right) } + \mu_{\ell} \left( \frac{n \bar{x}}{\sigma_{\ell}^2} + \frac{\mu_0}{\sigma_0^2} \right) \right\} } \times C_2 d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{\mu_{\ell}^2 }{2 \color{group5} \sigma_n^2 } + \mu_{\ell} \left( \frac{n \bar{x}}{\sigma_{\ell}^2} + \frac{\mu_0}{\sigma_0^2} \right) \right\} } \times C_2 d\mu_{\ell}. $$

Before completing the square, we want to factor out so that the exponential has the form of a Gaussian. However, to do this the factor needs to appear somewhere in the second term, so we introduce it the following way:

$$ \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{ {\color{group2} \mu_{\ell}^2} }{2 \color{group5} \sigma_n^2 } + {\color{group2} \mu_{\ell}} \left( \frac{n \bar{x}}{\color{group1} \sigma_{\ell}^2} + \frac{\mu_0}{\color{group6} \sigma_0^2} \right) \right\} } \times C_2 d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{\color{group2} \mu_{\ell}^2 }{2 \color{group5} \sigma_n^2 } + {\color{group2} \mu_{\ell}} \left( {\color{group5} \frac{1}{\sigma_n^2}} \right) \left( {\color{group5} \frac{\sigma_n^2}{1}} \right) \left( \frac{n \bar{x}}{\color{group1} \sigma_{\ell}^2} {\color{group6} \frac{\sigma_0^2}{\sigma_0^2}} + \frac{\mu_0}{\color{group6} \sigma_0^2} {\color{group1} \frac{\sigma_{\ell}^2}{\sigma_{\ell}^2}} \right) \right\} } \times C_2 d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{\mu_{\ell}^2 }{2 \sigma_n^2 } + \left( \frac{\mu_{\ell}}{\sigma_n^2} \right) \left( \frac{\cancel{\sigma_{\ell}^2 \sigma_0^2} }{n \sigma_0^2 + \sigma_{\ell}^2} \right) \left( \frac{n \bar{x} {\color{group6} \sigma_0^2} + \mu_0 {\color{group1} \sigma_{\ell}^2} }{\cancel{ {\color{group6} \sigma_0^2} {\color{group1} \sigma_{\ell}^2} } } \right) \right\} } \times C_2 d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{\mu_{\ell}^2}{2 \sigma_n^2} + \frac{\mu_{\ell}}{\sigma_n^2} \left( \frac{ n \bar{x} \sigma_0^2 + \mu_0 \sigma_{\ell}^2 }{n \sigma_0^2 + \sigma_{\ell}^2 } \right) \right\} } \times C_2 d\mu_{\ell}. $$
Equation 5: Complete the square.

Letting

$$ \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{\mu_{\ell}^2}{2 \sigma_n^2} + \frac{\mu_{\ell}}{\sigma_n^2} {\color{group5} \left( \frac{n \bar{x} \sigma_0^2 + \mu_0 \sigma_{\ell}^2 }{n \sigma_0^2 + \sigma_{\ell}^2 } \right) } \right\} } \times C_2 d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{\mu_{\ell}^2}{2 \sigma_n^2} + \frac{\mu_{\ell} {\color{group5} \mu_n}}{\sigma_n^2} \right\} } \times C_2 d\mu_{\ell}. $$

Now, we can complete the square by introducing into the exponent:

$$ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{\mu_{\ell}^2}{2 \sigma_n^2} + \frac{\mu_{\ell} \mu_n}{\sigma_n^2} + {\color{group5} 0} \right\} } \times C_2 d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{\mu_{\ell}^2}{2 \sigma_n^2} + \frac{\mu_{\ell} \mu_n}{\sigma_n^2} {\color{group5} - \frac{\mu_n^2}{2 \sigma_n^2} } {\color{group6} +} {\color{group5} \frac{\mu_n^2}{2 \sigma_n^2} } \right\} } \times C_2 d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{\mu_{\ell}^2}{2 \sigma_n^2} + \frac{\mu_{\ell} \mu_n}{\sigma_n^2} - \frac{\mu_n^2}{2 \sigma_n^2} \right\} } {\color{group6} \times} \exp{ \left\{ \frac{\mu_n^2}{2 \sigma_n^2} \right\} } \times C_2 d\mu_{\ell}. $$

Letting and factoring from the quadratic expression in the exponent,

$$ \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{\mu_{\ell}^2}{2 \sigma_n^2} + \frac{\mu_{\ell} \mu_n}{\sigma_n^2} - \frac{\mu_n^2}{2 \sigma_n^2} \right\} } {\color{group5} \exp{ \left\{ \frac{\mu_n^2}{2 \sigma_n^2} \right\} } \times C_2 } d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{\mu_{\ell}^2}{\color{group1} 2 \sigma_n^2} + \frac{-2 \mu_{\ell} \mu_n }{\color{group1} -2 \sigma_n^2 } - \frac{\mu_n^2}{\color{group1} 2 \sigma_n^2} \right\} } \times {\color{group5} C_3} d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ {\color{group1} - \frac{1}{2 \sigma_n^2}} \left[ \mu_{\ell}^2 - 2 \mu_{\ell} \mu_n - \mu_n^2 \right] \right\} } \times C_3 d\mu_{\ell}. $$
Equation 6: Solve the integral.

Since and are constant with respect to , we can move them outside of the integral:

$$ \int_{-\infty}^{+\infty} {\color{group5} C_1} \times \exp{ \left\{ - \frac{1}{2 \sigma_n^2} \left[ \mu_{\ell}^2 - 2 \mu_{\ell} \mu_n - \mu_n^2 \right] \right\} } \times {\color{group5} C_3} d\mu_{\ell} \\ = {\color{group5} C_1} \times {\color{group5} C_3} \times \int_{-\infty}^{+\infty} \times \exp{ \left\{ - \frac{1}{2 \sigma_n^2} \left[ \mu_{\ell}^2 - 2 \mu_{\ell} \mu_n - \mu_n^2 \right] \right\} } d\mu_{\ell}. $$

Since all probability density functions must integrate to 1, we know . Multiplying both sides by the normalizing constant shows us that the integral of the exponential is equal to the normalizing constant:

$$ 1 = \int_{- \infty}^{+ \infty} \univariateGaussianEQ{x}{\mu}{\sigma^2} d\mu \\ 1 \times {\color{group5} \sqrt{2 \pi \sigma^2}} = \cancel{\color{group5} \sqrt{2 \pi \sigma^2}} \frac{1}{\cancel{\sqrt{2 \pi \sigma^2}}} \int_{- \infty}^{+ \infty} \exp{\left\{ \frac{-1}{2\sigma^2} (x - \mu)^2 \right\}} d\mu \\ \sqrt{2 \pi {\color{group1} \sigma^2}} = \int_{-\infty}^{+ \infty} \exp{ \left\{ \frac{-1}{2 {\color{group1} \sigma^2}} (x - \mu)^2 \right\} } d\mu. $$

We can solve our integral in the same way:

$$ C_1 \times C_3 \times \int_{-\infty}^{+\infty} \times \exp{ \left\{ - \frac{1}{\color{group1} 2 \sigma_n^2} {\color{group2} \left[ \mu_{\ell}^2 - 2 \mu_{\ell} \mu_n - \mu_n^2 \right] } \right\} } d\mu_{\ell} \\ = C_1 \times C_3 \times \int_{-\infty}^{+\infty} \times \exp{ \left\{ - \frac{1}{\color{group1} 2 \sigma_n^2} {\color{group2} \left( \mu_{\ell} - \mu_n \right)^2 } \right\} } d\mu_{\ell} \\ = C_1 \times C_3 \times \sqrt{2 \pi {\color{group1} \sigma_n^2}}. $$
Equation 7: Reintroduce the constants .

Plugging in, we can simplify a bit:

$$ {\color{group1}C_1} \times {\color{group5} C_3} \times {\color{group6} \sqrt{2 \pi \sigma_n^2}} \\ = {\color{group1} \left( \frac{1}{\sqrt{2 \pi \sigma_{\ell}^2}} \right)^n \times \frac{1}{\cancel{\sqrt{2 \pi \sigma_0^2}}} } \times {\color{group5} \exp{ \left\{ \frac{\mu_n^2}{2 \sigma_n^2} \right\} } \times C_2 } \times {\color{group6} \sqrt{\frac{ \cancel{2 \pi} \sigma_{\ell}^2 \cancel{\sigma_0^2} }{n \sigma_0^2 + \sigma_{\ell}^2} } }. $$

Reintroducing , we have

$$ \frac{1}{\left( \sqrt{2 \pi \sigma_{\ell}^2} \right)^n} \times \sqrt{ \frac{\sigma_{\ell}^2 }{n \sigma_0^2 + \sigma_{\ell}^2 } } \times \exp{ \left\{ \frac{1}{2 \sigma_n^2} {\color{group2} \mu_n^2} \right\} } \times C_2 \\ = \frac{1}{\left( \sqrt{2 \pi \sigma_{\ell}^2} \right)^n} \times \sqrt{ \frac{\sigma_{\ell}^2 }{n \sigma_0^2 + \sigma_{\ell}^2 } } \times \exp{ \left\{ \frac{1}{2 {\color{group1} \sigma_n^2}} {\color{group2} \left( \frac{n \bar{x} \sigma_0^2 + \mu_0 \sigma_{\ell}^2 }{n \sigma_0^2 + \sigma_{\ell}^2 } \right)^2 } \right\} } \times C_2 \\ = \frac{1}{\left( \sqrt{2 \pi \sigma_{\ell}^2} \right)^n} \times \sqrt{ \frac{\sigma_{\ell}^2 }{n \sigma_0^2 + \sigma_{\ell}^2 } } \times \exp{ \left\{ \frac{ \cancel{\color{group1} n \sigma_0^2 + \sigma_{\ell}^2} {\color{group2} \left( n \bar{x} \sigma_0^2 + \mu_0 \sigma_{\ell}^2 \right)^2 } }{2 {\color{group1} \sigma_{\ell}^2 \sigma_0^2} \color{group2} \left( n \sigma_0^2 + \sigma_{\ell}^2 \right)^{\cancel{2}} } \right\} } \times C_2 \\ = \frac{1}{\left(\sqrt{2 \pi \sigma_{\ell}^2} \right)^n} \times \sqrt{ \frac{\sigma_{\ell}^2 }{n \sigma_0^2 + \sigma_{\ell}^2 } } \times \exp{ \left\{ \frac{ {\color{group2} (n \bar{x} \sigma_0^2)^2 + 2 (n \bar{x} \sigma_0^2) (\mu_0 \sigma_{\ell}^2) + (\mu_0 \sigma_{\ell}^2)^2 } }{2 {\color{group1} \sigma_{\ell}^2 \sigma_0^2} \color{group2} (n \sigma_0^2 + \sigma_{\ell}^2) } \right\} } \times C_2. $$

Dividing through the the numerator of the exponent, we obtain

$$ \frac{1}{\left(\sqrt{2 \pi \sigma_{\ell}^2} \right)^n} \times \sqrt{ \frac{\sigma_{\ell}^2 }{n \sigma_0^2 + \sigma_{\ell}^2 } } \times \exp{ \left\{ \frac{ {\color{group5} n^2 \bar{x}^2 (\sigma_0^2)^{\cancel{2}}} + 2 {\color{group5} (n \bar{x} \cancel{\sigma_0^2})} {\color{group6} (\mu_0 \cancel{\sigma_{\ell}^2})} + {\color{group6} \mu_0^2 (\sigma_{\ell}^2)^{\cancel{2}}} }{2 \cancel{\sigma_{\ell}^2 \sigma_0^2} (n \sigma_0^2 + \sigma_{\ell}^2) } \right\} } \times C_2 \\ = \frac{1}{\left(\sqrt{2 \pi \sigma_{\ell}^2} \right)^n} \times \sqrt{ \frac{\sigma_{\ell}^2 }{n \sigma_0^2 + \sigma_{\ell}^2 } } \times \exp{ \left\{ \frac{ {\color{group5} \frac{n^2 \bar{x}^2 \sigma_0^2}{\sigma_{\ell}^2}} + 2 {\color{group5} (n \bar{x})} {\color{group6} (\mu_0)} + {\color{group6} \frac{\mu_0^2 \sigma_{\ell}^2}{\sigma_0^2}} }{2 (n \sigma_0^2 + \sigma_{\ell}^2) } \right\} } \times C_2 \\ = \frac{1}{\left(\sqrt{2 \pi \sigma_{\ell}^2} \right)^n} \times \sqrt{ \frac{\sigma_{\ell}^2 }{2 (n \sigma_0^2 + \sigma_{\ell}^2) } } \times \exp{ \left\{ \frac{ \left( {\color{group5} \frac{n \bar{x} \sigma_0}{\sigma_{\ell}}} + {\color{group6} \frac{\mu_0 \sigma_{\ell}}{\sigma_0}} \right)^2 }{2 (n \sigma_0^2 + \sigma_{\ell}^2) } \right\} } \times C_2. $$

Finally, reintroducing , we obtain

$$ = \frac{1}{\left(\sqrt{2 \pi \sigma_{\ell}^2} \right)^n} \times \sqrt{ \frac{\sigma_{\ell}^2 }{n \sigma_0^2 + \sigma_{\ell}^2 } } \times \exp{ \left\{ \frac{ \left( \frac{n \bar{x} \sigma_0}{\sigma_{\ell}} + \frac{\mu_0 \sigma_{\ell}}{\sigma_0} \right)^2 }{2 (n \sigma_0^2 + \sigma_{\ell}^2) } \right\} } \times {\color{group5} C_2} \\ = \frac{1}{\left(\sqrt{2 \pi \sigma_{\ell}^2} \right)^n} \times \sqrt{ \frac{\sigma_{\ell}^2 }{n \sigma_0^2 + \sigma_{\ell}^2 } } \times \exp{ \left\{ \frac{ \left( \frac{n \bar{x} \sigma_0}{\sigma_{\ell}} + \frac{\mu_0 \sigma_{\ell}}{\sigma_0} \right)^2 }{2 (n \sigma_0^2 + \sigma_{\ell}^2) } \right\} } {\color{group1} \times} {\color{group5} \exp{ \left\{ - \frac{\sum\limits_i^n x_i^2 }{2 \sigma_{\ell}^2 } - \frac{\mu_0^2 }{2 \sigma_0^2 } \right\} } } \\ = \sqrt{ \frac{\sigma_{\ell}^2 }{(n \sigma_0^2 + \sigma_{\ell}^2) (2 \pi \sigma_{\ell}^2)^n } } \times \exp{ \left\{ \frac{ \left( \frac{n \bar{x} \sigma_0}{\sigma_{\ell}} + \frac{\mu_0 \sigma_{\ell}}{\sigma_0} \right)^2 }{2 (n \sigma_0^2 + \sigma_{\ell}^2) } {\color{group1} -} \frac{\sum\limits_i^n x_i^2 }{2 \sigma_{\ell}^2 } - \frac{\mu_0^2 }{2 \sigma_0^2 } \right\} }. $$
Posterior on Likelihood Parameters
$$ \begin{align} \posterior{\Theta}{x_1, x_2, \dots, x_n} &= \frac{ \left[ \prod\limits_i^n \normalAbbreviation{x_i}{\mu_{\ell}}{\sigma_{\ell}^2} \right] \ \normalAbbreviation{\mu_{\ell}}{\mu_0}{\sigma_0^2} }{\int_{\mathfrak{D}(\mu_{\ell})} \left[ \prod\limits_i^n \normalAbbreviation{x_i}{\mu_{\ell}}{\sigma_{\ell}^2} \right] \ \normalAbbreviation{\mu_{\ell}}{\mu_0}{\sigma_0^2} d\mu_{\ell} } \tag{1} \label{normalKnownPosterior1} \\ &= \normalAbbreviation{ \mu_{\ell} }{\mu_n }{\color{group1} \sigma_n^2 } \tag{2} \label{normalKnownPosterior2} \\ &= \circleEquation{ \normalAbbreviation{ \mu_{\ell} }{\mu_n }{\color{group1} \frac{\sigma_{\ell}^2 }{n + \kappa_0 } } } \tag{3} \label{normalKnownPosterior3} \\ \end{align} $$
Equation 1: Plug in using the result in Equation 1 of the Marginal Likelihood derivation.

Recall Step 4 of the approach and plug in the same way we did in Equation 1 of the Marginal Likelihood derivation:

$$ \posterior{\Theta}{x_1, x_2, \dots, x_n} = \frac{\likelihood{x_1, x_2, \dots, x_n}{\Theta} \ \prior{\Theta}} {\marginalLikelihood{x_1, x_2, \dots, x_n}} \\ = \frac{ {\color{group1} \likelihood{x_1, x_2, \dots, x_n}{\Theta}} \ {\color{group2} \prior{\Theta}} }{\int_{\mathfrak{D}(\Theta)} {\color{group1} \likelihood{x_1, x_2, \dots, x_n}{\Theta}} \ {\color{group2} \prior{\Theta}} d{\Theta} } \\ = \frac{ {\color{group1} \left[ \prod\limits_i^n \normalAbbreviation{x_i}{\mu_{\ell}}{\sigma_{\ell}^2} \right] } \ {\color{group2} \normalAbbreviation{\mu_{\ell}}{\mu_0}{\sigma_0^2} } }{\int_{\mathfrak{D}(\Theta)} {\color{group1} \left[ \prod\limits_i^n \normalAbbreviation{x_i}{\mu_{\ell}}{\sigma_{\ell}^2} \right] } \ {\color{group2} \normalAbbreviation{\mu_{\ell}}{\mu_0}{\sigma_0^2} } d\Theta }. $$

Again, for the Normal-Known model, the likelihood variance is constant (known with probability 1), meaning there is no prior on the likelihood variance. The only unknown variable is the likelihood mean ; hence,

$$ \frac{ \left[ \prod\limits_i^n \normalAbbreviation{x_i}{\mu_{\ell}}{\sigma_{\ell}^2} \right] \ \normalAbbreviation{\mu_{\ell}}{\mu_0}{\sigma_0^2} }{\int_{\color{group5} \mathfrak{D}(\Theta)} \left[ \prod\limits_i^n \normalAbbreviation{x_i}{\mu_{\ell}}{\sigma_{\ell}^2} \right] \ \normalAbbreviation{\mu_{\ell}}{\mu_0}{\sigma_0^2} {\color{group5} d\Theta} } = \frac{ \left[ \prod\limits_i^n \normalAbbreviation{x_i}{\mu_{\ell}}{\sigma_{\ell}^2} \right] \ \normalAbbreviation{\mu_{\ell}}{\mu_0}{\sigma_0^2} }{\int_{\color{group5} - \infty}^{\color{group5} + \infty} \left[ \prod\limits_i^n \normalAbbreviation{x_i}{\mu_{\ell}}{\sigma_{\ell}^2} \right] \ \normalAbbreviation{\mu_{\ell}}{\mu_0}{\sigma_0^2} {\color{group5} d\mu_{\ell}} }. $$
Equation 2: Recall Equations 5 and 6 of the Marginal Likelihood derivation.

The denominator is exactly what we derived in Equation 5 of the Marginal Likelihood derivation, and the numerator is exactly the same thing without the integration. Plugging in that result, factoring the quadratic expressions, and moving constants to the front, we have the following:

$$ \frac{ C_1 \times \exp{ \left\{ - \frac{1}{2 {\color{group1} \sigma_n^2}} \left[ {\color{group6} \mu_{\ell}^2} - 2 {\color{group6} \mu_{\ell}} {\color{group2} \mu_n} + {\color{group2} \mu_n}^2 \right] \right\} } \times C_3 }{\int_{-\infty}^{+\infty} C_1 \times \exp{ \left\{ - \frac{1}{2 {\color{group1} \sigma_n^2}} \left[ {\color{group6} \mu_{\ell}^2} - 2 {\color{group6} \mu_{\ell}} {\color{group2} \mu_n} + {\color{group2} \mu_n}^2 \right] \right\} } \times C_3 d\mu_{\ell} } \\ = \frac{ C_1 \times C_3 \times \exp{ \left\{ - \frac{1}{2 {\color{group1} \sigma_n^2}} \left( {\color{group6} \mu_{\ell}} - {\color{group2} \mu_n} \right)^2 \right\} } }{C_1 \times C_3 \times \int_{-\infty}^{+\infty} \exp{ \left\{ - \frac{1}{2 {\color{group1} \sigma_n^2}} \left( {\color{group6} \mu_{\ell}} - {\color{group2} \mu_n} \right)^2 \right\} } \times d\mu_{\ell} }. $$

Recall that we solved the denominator’s integral in Equation 6 of the marginal likelihood derivation. Plugging that solution in, we obtain

$$ \frac{ \cancel{C_1} \times \cancel{C_3} \times \exp{ \left\{ - \frac{1}{2 \sigma_n^2} \left( \mu_{\ell} - \mu_n \right)^2 \right\} } }{\cancel{C_1} \times \cancel{C_3} \times {\color{group5} \int_{-\infty}^{+\infty} \exp{ \left\{ - \frac{1}{2 \sigma_n^2} \left( \mu_{\ell} - \mu_n \right)^2 \right\} } } \times d\mu_{\ell} } \\ = \frac{ \exp{ \left\{ - \frac{1}{2 \sigma_n^2} \left( \mu_{\ell} - \mu_n \right)^2 \right\} } }{\color{group5} \sqrt{2 \pi \sigma_n^2} } \\ = \normalAbbreviation{\mu_{\ell}}{\mu_n}{\sigma_n^2}. $$
Equation 3: Simplify the covariance to show the posterior has the same form as the prior.

Recalling that we defined the variance of the prior as . Plugging in, we can simplify as follows:

$$ \begin{align} \sigma_n^2 &= \frac{ \sigma_{\ell}^2 {\color{group1} \sigma_0^2} }{n {\color{group1} \sigma_0^2} + \sigma_{\ell}^2 } = \frac{ \sigma_{\ell}^2 \times {\color{group1} \frac{\sigma_{\ell}^2}{\kappa_0}} }{\sigma_{\ell}^2 + n \color{group1} \frac{\sigma_{\ell}^2}{\kappa_0} } \\ &= \frac{ \frac{ \sigma_{\ell}^2 {\color{group1} \sigma_{\ell}^2} }{\cancel{\color{group1} \kappa_0} } }{\frac{ n {\color{group1} \sigma_{\ell}^2} + {\color{group1} \kappa_0} \sigma_{\ell}^2 }{\cancel{\color{group1} \kappa_0} } } = \frac{\sigma_{\ell}^2 \sigma_{\ell}^2 }{n \sigma_{\ell}^2 + \kappa_0 \sigma_{\ell}^2 } \\ &= \frac{\sigma_{\ell}^2 \cancel{\sigma_{\ell}^2} }{n \cancel{\sigma_{\ell}^2} + \kappa_0 \cancel{\sigma_{\ell}^2} } = \frac{\sigma_{\ell}^2 }{n + \kappa_0 }. \end{align} $$
Posterior Predictive of Test Data
$$ \begin{align} \posteriorPredictive{x_*}{x_1, x_2, \dots, x_n} &= \int_{-\infty}^{+\infty} \normalAbbreviation{x_*}{\mu_{\ell}}{\color{group1} \sigma_{\ell}^2} \normalAbbreviation{\mu_{\ell} }{\color{group2} \mu_n }{\color{group1} \sigma_n^2 } d\mu_{\ell} \tag{1} \label{normalKnownPosteriorPredictive1}\\ &= \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ \frac{ \sigma_n^2 \left( x_*^2 - 2 \mu_{\ell} x_* + \mu_{\ell}^2 \right) + \sigma_{\ell}^2 \left( \mu_{\ell}^2 - 2 \mu_{\ell} \mu_n + \mu_n^2 \right) }{-2 \sigma_{\ell}^2 \sigma_n^2 } \right\} } d\mu_{\ell} \tag{2} \label{normalKnownPosteriorPredictive2}\\ &= \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ \frac{ \sigma_*^2 \left[ \mu_{\ell}^2 - \frac{ -2 \mu_{\ell} \left( \sigma_{\ell}^2 x_* + \sigma_{\ell}^2 \mu_n \right) }{\sigma_*^2 } - \frac{ \sigma_{\ell}^2 x_*^2 + \sigma_{\ell}^2 \mu_n^2 }{\sigma_*^2 } \right] }{-2 \sigma_{\ell}^2 \sigma_n^2 } \right\} } d\mu_{\ell} \tag{3} \label{normalKnownPosteriorPredictive3}\\ &= \int_{-\infty}^{+\infty} C_4 \times C_5 \times \exp{ \left\{ \frac{ - \sigma_*^2 }{2 \sigma_{\ell}^2 \sigma_n^2 } \left[ \mu_{\ell} - \frac{ \left( \sigma_n^2 x + \sigma_{\ell}^2 \mu_n \right) }{\sigma_*^2 } \right]^2 \right\} } d\mu_{\ell} \tag{4} \label{normalKnownPosteriorPredictive4}\\ &= C_5 \times \frac{ 1 }{\sqrt{2 \pi \sigma_*^2} } \tag{5} \label{normalKnownPosteriorPredictive5}\\ &= \frac{ 1 }{\sqrt{2 \pi \sigma_*^2} } \times \exp{ \left\{ \frac{ 1 }{- 2 \sigma_{\ell}^2 \sigma_n^2 {\color{group1} \sigma_*^2} } \left( \sigma_*^2 \left[ \sigma_n^2 x_*^2 + \sigma_{\ell}^2 \mu_n^2 \right] - \left[ \sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n \right]^2 \right) \right\} } \tag{6} \label{normalKnownPosteriorPredictive6}\\ &= \circleEquation{ \normalAbbreviation{x_* }{\color{group2} \mu_n }{\color{group1} \sigma_{\ell}^2 + \sigma_n^2 } } \tag{7} \label{normalKnownPosteriorPredictive7}\\ \end{align} $$
Equation 1: Plug into the definition of posterior predictive.

As I stated in the approach, the posterior predictive is equal to

$$ \int_{\color{group5} \mathfrak{D}(\Theta)} {\color{group1} \likelihood{x_*}{\Theta}} \ {\color{group2} \posterior{\Theta}{x_1, x_2, \dots, x_n}} d{\color{group5} \Theta}. $$

From the defintion of the model, we know the likelihood is , and in the earlier section, we derived the posterior, So, plugging in we have

$$ \int_{\color{group5} - \infty}^{\color{group5} + \infty} {\color{group1} \normalAbbreviation{x_*}{\mu_{\ell}}{\sigma_{\ell}^2} } {\color{group2} \normalAbbreviation{\mu_{\ell}}{\mu_n}{\sigma_n^2} } d{\color{group5} \mu_{\ell}}. $$

Again, we integrate on the interval because is a real number such that .

Equation 2: Substitute for the normalizing constant and expand the exponent.

Plugging in the Gaussian expressions, moving the normalizing constants to the front, and simplifying the exponents, we start with

$$ \int_{-\infty}^{+\infty} {\color{group5} \normalAbbreviation{x_*}{\mu_{\ell}}{\sigma_{\ell}^2}} {\color{group6} \normalAbbreviation{\mu_{\ell}}{\mu_n}{\sigma_n^2}} d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} {\color{group5} \frac{1}{\sqrt{(2 \pi \sigma_{\ell}^2)}}} \times {\color{group6} \frac{1}{\sqrt{(2 \pi \sigma_n^2)}}} \times {\color{group5} \exp{ \left\{ \frac{-1}{2 \sigma_{\ell}^2} (x_* - \mu_{\ell})^2 \right\} } } {\color{group6} \exp{ \left\{ \frac{-1}{2 \sigma_n^2} (\mu_{\ell} - \mu_n)^2 \right\} } } d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} \frac{1}{\sqrt{ {\color{group5} (2\pi\sigma_{\ell}^2)} {\color{group6} (2\pi\sigma_n^2)}} } \times \exp{ \left\{ {\color{group5} \frac{-1}{2 \sigma_{\ell}^2} (x_* - \mu_{\ell})^2 } - {\color{group6} \frac{1}{2 \sigma_n^2} (\mu_{\ell} - \mu_n)^2 } \right\} } d\mu_{\ell}. $$

Letting , and expanding both and ,

$$ \int_{-\infty}^{+\infty} {\color{group1} C_4} \times \exp{ \left\{ {\color{group5} \frac{-1}{2 \sigma_{\ell}^2} (x_*^2 - 2x_*\mu_{\ell} + \mu_{\ell}^2) } - {\color{group6} \frac{1}{2 \sigma_n^2} (\mu_{\ell}^2 - 2\mu_{\ell}\mu_n + \mu_n^2) } \right\} } d\mu_{\ell}. $$

Obtain a common denominator by introducing and to the first and second terms, respectively:

$$ \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ {\color{group5} \frac{-1}{2 \sigma_{\ell}^2}} \left( {\color{group6} \frac{\sigma_n^2}{\sigma_n^2}} \right) {\color{group5} (x_*^2 - 2x_*\mu_{\ell} + \mu_{\ell}^2)} - {\color{group6} \frac{1}{2 \sigma_n^2}} \left( {\color{group5} \frac{\sigma_{\ell}^2}{\sigma_{\ell}^2}} \right) {\color{group6} (\mu_{\ell}^2 - 2\mu_{\ell}\mu_n + \mu_n^2)} \right\} } d\mu \\ = \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ \frac{- \color{group6} \sigma_n^2 }{ {\color{group5} 2 \sigma_{\ell}^2} {\color{group6} \sigma_n^2} } {\color{group5} (x_*^2 - 2x_*\mu_{\ell} + \mu_{\ell}^2)} - \frac{\color{group5} \sigma_{\ell}^2 }{ {\color{group6} 2 \sigma_n^2} {\color{group5} \sigma_{\ell}^2} } {\color{group6} (\mu_{\ell}^2 - 2\mu_{\ell}\mu_n + \mu_n^2)} \right\} } d\mu $$

Factoring ,

$$ \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ \frac{- \sigma_n^2 }{\color{group2} 2 \sigma_{\ell}^2 \sigma_n^2 } (x_*^2 - 2x_*\mu_{\ell} + \mu_{\ell}^2) - \frac{\sigma_{\ell}^2 }{\color{group2} 2 \sigma_n^2 \sigma_{\ell}^2 } (\mu_{\ell}^2 - 2\mu_{\ell}\mu_n + \mu_n^2) \right\} } d\mu \\ = \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ \frac{ 1 }{\color{group2} - 2 \sigma_{\ell}^2 \sigma_n^2 } \left[ \sigma_n^2 \left( x_*^2 - 2 \mu x_* + \mu^2 \right) + \sigma_{\ell}^2 \left( \mu^2 - 2 \mu \mu_n + \mu_n^2 \right) \right] \right\} } d\mu \\ = \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ \frac{ \sigma_n^2 \left( x_*^2 - 2 \mu x_* + \mu^2 \right) + \sigma_{\ell}^2 \left( \mu^2 - 2 \mu \mu_n + \mu_n^2 \right) }{\color{group2} - 2 \sigma_{\ell}^2 \sigma_n^2 } \right\} } d\mu. $$
Equation 3: Factor out .

Distributing the posterior and likelihood variances,

$$ \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ \frac{ \sigma_n^2 \left( x_*^2 {\color{group2} - 2 \mu_{\ell}} x_* + {\color{group1} \mu_{\ell}^2} \right) + \sigma_{\ell}^2 \left( \mu_{\ell}^2 {\color{group2} - 2 \mu_{\ell}} \mu_n + {\color{group1} \mu_n^2} \right) }{-2 \sigma_{\ell}^2 \sigma_n^2 } \right\} } d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ \frac{ x_*^2 \sigma_n^2 - {\color{group2} 2 \mu_{\ell}} x_* \sigma_n^2 + {\color{group1} \mu_{\ell}^2} \sigma_n^2 + {\color{group1} \mu_{\ell}^2} \sigma_{\ell}^2 - {\color{group2} 2 \mu_{\ell}} \mu_n \sigma_{\ell}^2 + \mu_n^2 \sigma_{\ell}^2 }{-2 \sigma_{\ell}^2 \sigma_n^2 } \right\} } d\mu_{\ell}. $$

Grouping like terms in the exponent,

$$ \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ \frac{ {\color{group1} \mu_{\ell}^2} \sigma_n^2 + {\color{group1} \mu_{\ell}^2} \sigma_{\ell}^2 - {\color{group2} 2 \mu_{\ell}} x_* \sigma_n^2 - {\color{group2} 2 \mu_{\ell}} \mu_n \sigma_{\ell}^2 + \mu_n^2 \sigma_{\ell}^2 + x_*^2 \sigma_n^2 }{-2 \sigma_{\ell}^2 \sigma_n^2 } \right\} } d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ \frac{ {\color{group1} \mu_{\ell}^2} (\sigma_n^2 + \sigma_{\ell}^2) - {\color{group2} 2 \mu_{\ell}} (x_* \sigma_n^2 + \mu_n \sigma_{\ell}^2) + \mu_n^2 \sigma_{\ell}^2 + x_*^2 \sigma_n^2 }{-2 \sigma_{\ell}^2 \sigma_n^2 } \right\} } d\mu_{\ell}. $$

Letting , we have

$$ \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ \frac{ {\color{group1} \mu_{\ell}^2} {\color{group6} \sigma_*^2} - {\color{group2} 2 \mu_{\ell}} (x_* \sigma_n^2 + \mu_n \sigma_{\ell}^2) + \mu_n^2 \sigma_{\ell}^2 + x_*^2 \sigma_n^2 }{-2 \sigma_{\ell}^2 \sigma_n^2 } \right\} } d\mu_{\ell}. $$

Factor from the numberator’s exponent to obtain

$$ \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ \frac{ {\color{group6} \sigma_*^2} \left[ {\color{group1} \mu_{\ell}^2} - \frac{2 {\color{group2} \mu_{\ell}} \left( \sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n \right) }{\color{group6} \sigma_*^2 } + \frac{\sigma_n^2 x_*^2 + \sigma_{\ell}^2 \mu_n^2 }{\color{group6} \sigma_*^2 } \right] }{-2 \sigma_{\ell}^2 \sigma_n^2 } \right\} } d\mu_{\ell}. $$
Equation 4: Complete the square.

To complete the square, we introduce to the numerator in the exponent, and let , which, of course, is constant with respect to .

$$ \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ \frac{ {\color{group6} \sigma_*^2} \left[ \mu_{\ell}^2 - \frac{ 2 \mu_{\ell} \left( \sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n \right) }{\sigma_*^2 } {\color{group5} + 0} + \frac{\sigma_n^2 x_*^2 + \sigma_{\ell}^2 \mu_n^2 }{\sigma_*^2 } \right] }{\color{group6} - 2 \sigma_{\ell}^2 \sigma_n^2 } \right\} } d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ \frac{ {\color{group6} \sigma_*^2} \left[ \mu_{\ell}^2 - \frac{ 2 \mu_{\ell} \left( \sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n \right) }{\sigma_*^2 } {\color{group5} + \left( \frac{(\sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n) }{\sigma_*^2 } \right)^2 - \left( \frac{(\sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n) }{\sigma_*^2 } \right)^2 } + \frac{\sigma_n^2 x_*^2 + \sigma_{\ell}^2 \mu_n^2 }{\sigma_*^2 } \right] }{\color{group6} - 2 \sigma_{\ell}^2 \sigma_n^2 } \right\} } d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ \frac{ {\color{group6} \sigma_*^2} \left[ \mu_{\ell}^2 - \frac{ 2 \mu_{\ell} \left( \sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n \right) }{\sigma_*^2 } {\color{group5} + \left( \frac{(\sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n) }{\sigma_*^2 } \right)^2 } \right] }{\color{group6} - 2 \sigma_{\ell}^2 \sigma_n^2 } {\color{group1} +} \frac{ {\color{group6} \sigma_*^2} \left[ {\color{group5} - \left( \frac{(\sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n) }{\sigma_*^2 } \right)^2 } + \frac{\sigma_n^2 x_*^2 + \sigma_{\ell}^2 \mu_n^2 }{\sigma_*^2 } \right] }{\color{group6} - 2 \sigma_{\ell}^2 \sigma_n^2 } \right\} } d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ \frac{ {\color{group6} \sigma_*^2} \left[ \mu_{\ell}^2 - \frac{ 2 \mu_{\ell} \left( \sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n \right) }{\sigma_*^2 } {\color{group5} + \left( \frac{(\sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n) }{\sigma_*^2 } \right)^2 } \right] }{\color{group6} - 2 \sigma_{\ell}^2 \sigma_n^2 } \right\} } {\color{group1} \times} C_5 d\mu_{\ell}. $$

Factoring the quadratic expression in the exponent, we obtain the following

$$ \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ \frac{ {\color{group6} \sigma_*^2} \left[ {\color{group2} \mu_{\ell}}^2 - \frac{ 2 {\color{group2} \mu_{\ell}} {\color{group1} \left( \sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n \right) } }{\color{group1} \sigma_*^2 } + \left( {\color{group1} \frac{(\sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n) }{\sigma_*^2 } } \right)^2 \right] }{\color{group6} - 2 \sigma_{\ell}^2 \sigma_n^2 } \right\} } \times C_5 d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} C_4 \times \exp{ \left\{ \frac{ {\color{group6} \sigma_*^2} }{\color{group6} - 2 \sigma_{\ell}^2 \sigma_n^2 } \left[ {\color{group2} \mu_{\ell}} - \left( {\color{group1} \frac{(\sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n) }{\sigma_*^2 } } \right) \right]^2 \right\} } \times C_5 d\mu_{\ell}. $$
Equation 5: Rewrite the normalizing constant so the integral evaluates to 1.

Introduce into , so that we have the corresponding normalizing constant for the gaussian expression being integrated. Recall that :

$$ \int_{-\infty}^{+\infty} {\color{group5} C_4} \times \exp{ \left\{ \frac{ \sigma_*^2 }{- 2 \sigma_{\ell}^2 \sigma_n^2 } \left[ \mu_{\ell} - \left( \frac{(\sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n) }{\sigma_*^2 } \right) \right]^2 \right\} } \times C_5 d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} {\color{group5} \frac{1}{\sqrt{(2\pi\sigma_{\ell}^2)(2\pi\sigma_n^2)}} } \times \exp{ \left\{ \frac{ \sigma_*^2 }{- 2 \sigma_{\ell}^2 \sigma_n^2 } \left[ \mu_{\ell} - \left( \frac{(\sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n) }{\sigma_*^2 } \right) \right]^2 \right\} } \times C_5 d\mu_{\ell} \\ = \int_{-\infty}^{+\infty} {\color{group6} \sqrt{\frac{\sigma_*^2}{\sigma_*^2}} } \times {\color{group5} \frac{1}{\sqrt{(2\pi\sigma_{\ell}^2)(2\pi\sigma_n^2)}} } \times \exp{ \left\{ \frac{ {\color{group6} \sigma_*^2} }{- 2 {\color{group5} \sigma_{\ell}^2 \sigma_n^2} } \left[ \mu_{\ell} - \left( \frac{(\sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n) }{\sigma_*^2 } \right) \right]^2 \right\} } \times C_5 d\mu_{\ell} $$

Moving the appropriate constants outside of the integral, we can make the integral evaluate to 1:

$$ \int_{-\infty}^{+\infty} \sqrt{\frac{\color{group1} \sigma_*^2}{\color{group2} \sigma_*^2}} \times \frac{1}{\sqrt{ ({\color{group2} 2 \pi} {\color{group1} \sigma_{\ell}^2)} ({\color{group1} 2 \pi \sigma_n^2})} } \times \exp{ \left\{ {\color{group1} \frac{ \sigma_*^2 }{- 2 \sigma_{\ell}^2 \sigma_n^2 } } \left[ \mu_{\ell} - \left( \frac{(\sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n) }{\sigma_*^2 } \right) \right]^2 \right\} } \times {\color{group2} C_5} d\mu_{\ell} \\ = {\color{group2} C_5 \times \frac{ 1 }{\sqrt{2 \pi \sigma_*^2} } } \int_{-\infty}^{+\infty} {\color{group1} \sqrt{ \frac{ \sigma_*^2 }{2 \pi \sigma_0^2 \sigma_n^2 } } } \exp{ \left\{ {\color{group1} \frac{ - \sigma_*^2 }{2 \sigma_0^2 \sigma_n^2 } } \left[ \mu_{\ell} - \frac{ \left( \sigma_n^2 x + \sigma_0^2 \mu_n \right) }{\sigma_*^2 } \right]^2 \right\} } d\mu_{\ell} \\ = {\color{group2} C_5 \times \frac{ 1 }{\sqrt{2 \pi \sigma_*^2} } } \cancel{ \int_{-\infty}^{+\infty} \normalAbbreviation{ \mu_{\ell} }{\frac{ \left( \sigma_n^2 x + \sigma_0^2 \mu_n \right) }{\sigma_*^2 } }{\color{group1} \frac{ \sigma_0^2 \sigma_n^2 }{\sigma_*^2 } } d\mu_{\ell} } \\ = {\color{group2} C_5 \times \frac{ 1 }{\sqrt{2 \pi \sigma_*^2} } } \times 1. $$

Recall that the integral of a Gaussian evaluates to 1 when integrating along the entire real line.

Equation 6: Reintroduce the constant .

Recall that . Therefore,

$$ {\color{group5} C_5} \times \frac{1}{\sqrt{2 \pi \sigma_*^2}} = \frac{1}{\sqrt{2 \pi \sigma_*^2}} {\color{group5} \exp{ \left\{ \frac{\sigma_*^2}{2 \sigma_{\ell}^2 \sigma_n^2} \left[ - \left( \frac{(\sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n) }{\sigma_*^2 } \right)^2 + \frac{\sigma_n^2 x_*^2 + \sigma_{\ell}^2 \mu_n^2 }{\sigma_*^2 } \right] \right\} } }. $$

Rearranging the two terms in the exponent and simplifying:

$$ \frac{ 1 }{\sqrt{2 \pi \sigma_*^2} } \times \exp{ \left\{ \frac{ \sigma_*^2 }{- 2 \sigma_{\ell}^2 \sigma_n^2 } \left( \left[ {\color{group1} \frac{\sigma_n^2 x_*^2 + \sigma_{\ell}^2 \mu_n^2 }{\sigma_*^2 } } \right] - \left[ {\color{group2} \frac{\sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n }{\sigma_*^2 } } \right]^2 \right) \right\} } \\ = \frac{ 1 }{\sqrt{2 \pi \sigma_*^2} } \times \exp{ \left\{ \frac{ \cancel{\sigma_*^2} }{- 2 \sigma_{\ell}^2 \sigma_n^2 } \left( \frac{ 1 }{\color{group1} \cancel{\sigma_*^2} } {\color{group1} \left[ \sigma_n^2 x_*^2 + \sigma_{\ell}^2 \mu_n^2 \right] } - \frac{ 1 }{\color{group2} (\sigma_*^2)^{\cancel{2}} } {\color{group2} \left[ \sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n \right]^2 } \right) \right\} } \\ = \frac{ 1 }{\sqrt{2 \pi \sigma_*^2} } \times \exp{ \left\{ \frac{1}{- 2 \sigma_{\ell}^2 \sigma_n^2} \times \left( \left[ \sigma_n^2 x_*^2 + \sigma_{\ell}^2 \mu_n^2 \right] - \frac{ 1 }{\sigma_*^2 } \left[ \sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n \right]^2 \right) \right\} }. $$

Introduce the following way:

$$ \frac{ 1 }{\sqrt{2 \pi \sigma_*^2} } \times \exp{ \left\{ \frac{1}{- 2 \sigma_{\ell}^2 \sigma_n^2} \times {\color{group6} 1} \times \left( \left[ \sigma_n^2 x_*^2 + \sigma_{\ell}^2 \mu_n^2 \right] - \frac{ 1 }{\sigma_*^2 } \left[ \sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n \right]^2 \right) \right\} } \\ = \frac{ 1 }{\sqrt{2 \pi \sigma_*^2} } \times \exp{ \left\{ \frac{1}{- 2 \sigma_{\ell}^2 \sigma_n^2} \times {\color{group6} \frac{\sigma_*^2}{\sigma_*^2}} \times \left( \left[ \sigma_n^2 x_*^2 + \sigma_{\ell}^2 \mu_n^2 \right] - \frac{ 1 }{\sigma_*^2 } \left[ \sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n \right]^2 \right) \right\} } \\ = \frac{ 1 }{\sqrt{2 \pi \sigma_*^2} } \times \exp{ \left\{ \frac{1}{- 2 \sigma_{\ell}^2 \sigma_n^2 {\color{group6} \sigma_*^2}} \left( {\color{group6} \sigma_*^2} \left[ \sigma_n^2 x_*^2 + \sigma_{\ell}^2 \mu_n^2 \right] - \frac{ \cancel{\color{group6} \sigma_*^2} }{\cancel{\color{group6} \sigma_*^2} } \left[ \sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n \right]^2 \right) \right\} } \\ = \frac{ 1 }{\sqrt{2 \pi \sigma_*^2} } \times \exp{ \left\{ \frac{1}{- 2 \sigma_{\ell}^2 \sigma_n^2 \sigma_*^2} \left( \sigma_*^2 \left[ \sigma_n^2 x_*^2 + \sigma_{\ell}^2 \mu_n^2 \right] - \left[ \sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n \right]^2 \right) \right\} } \\ $$
Equation 7: Expand, and then simplify.

Since ,

$$ \begin{align} \frac{ 1 }{\sqrt{2 \pi \sigma_*^2} } \times \exp{ \left\{ \frac{ 1 }{- 2 \sigma_{\ell}^2 \sigma_n^2 \sigma_*^2 } \left( {\color{group2} \sigma_*^2} {\color{group5} \left[ \sigma_n^2 x_*^2 + \sigma_{\ell}^2 \mu_n^2 \right] } {\color{group6} - \left[ \sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n \right]^2 } \right) \right\} } &=\frac{ 1 }{\sqrt{2 \pi \sigma_*^2} } \times \exp{ \left\{ \frac{ 1 }{- 2 \sigma_{\ell}^2 \sigma_n^2 \sigma_*^2 } \left( {\color{group2} (\sigma_{\ell}^2 + \sigma_n^2)} {\color{group5} \left[ \sigma_n^2 x_*^2 + \sigma_{\ell}^2 \mu_n^2 \right] } {\color{group6} - \left[ \sigma_n^2 x_* + \sigma_{\ell}^2 \mu_n \right]^2 } \right) \right\} } \\ &=\frac{ 1 }{\sqrt{2 \pi \sigma_*^2} } \times \exp{ \left\{ \frac{-1}{2 \sigma_*^2} \frac{ \left[ \left( {\color{group2} \sigma_{\ell}^2} {\color{group5} \sigma_n^2 x_*^2} + {\color{group2} \sigma_{\ell}^2} {\color{group5} \sigma_{\ell}^2 \mu_n^2} \right) \right] }{\sigma_{\ell}^2 \sigma_n^2 } \right\} } \times \exp{ \left\{ \frac{-1}{2 \sigma_*^2} \frac{ \left[ \left( {\color{group2} \sigma_n^2} {\color{group5} \sigma_n^2 x_*^2} + {\color{group2} \sigma_n^2} {\color{group5} \sigma_{\ell}^2 \mu_n^2} \right) - {\color{group6} (\sigma_n^2 x_*)^2} - 2 {\color{group6} (\sigma_n^2 x_*) (\sigma_{\ell}^2 \mu_n) } - {\color{group6} (\sigma_{\ell}^2 \mu_n)^2} \right] }{\sigma_{\ell}^2 \sigma_n^2 } \right\} }. \end{align} $$

We can simplify further by breaking the fractions in the exponents apart:

$$ \frac{ 1 }{\sqrt{2 \pi \sigma_*^2} } \times \exp{ \left\{ \frac{-1}{2 \sigma_*^2} \frac{ \left[ {\color{group2} \sigma_{\ell}^2} {\color{group5} \sigma_n^2 x_*^2} + {\color{group2} \sigma_{\ell}^2} {\color{group5} \sigma_{\ell}^2 \mu_n^2} \right] }{\sigma_{\ell}^2 \sigma_n^2 } \right\} } \\ \times \exp{ \left\{ \frac{-1}{2 \sigma_*^2} \frac{ \left[ \left( {\color{group2} \sigma_n^2} {\color{group5} \sigma_n^2 x_*^2} + {\color{group2} \sigma_n^2} {\color{group5} \sigma_{\ell}^2 \mu_n^2} \right) - {\color{group6} (\sigma_n^2 x_*)^2} - 2 {\color{group6} (\sigma_n^2 x_*) (\sigma_{\ell}^2 \mu_n) } - {\color{group6} (\sigma_{\ell}^2 \mu_n)^2} \right] }{\sigma_{\ell}^2 \sigma_n^2 } \right\} } \\ = \frac{ 1 }{\sqrt{2 \pi \sigma_*^2} } \times \exp{ \left\{ \frac{-1}{2 \sigma_*^2} \left[ \frac{ {\color{group2} \cancel{\sigma_{\ell}^2}} {\color{group5} \cancel{\sigma_n^2} x_*^2} }{\cancel{\sigma_{\ell}^2 \sigma_n^2} } +\frac{ {\color{group2} \cancel{\sigma_{\ell}^2}} {\color{group5} \sigma_{\ell}^2 \mu_n^2} }{\cancel{\sigma_{\ell}^2} \sigma_n^2 } \right] \right\} } \\ \times \exp{ \left\{ \frac{-1}{2 \sigma_*^2} \left[ \frac{ {\color{group2} \cancel{\sigma_n^2}} {\color{group5} \sigma_n^2 x_*^2} }{\sigma_{\ell}^2 \cancel{\sigma_n^2} } + \frac{ {\color{group2} \cancel{\sigma_n^2}} {\color{group5} \cancel{\sigma_{\ell}^2} \mu_n^2} }{\cancel{\sigma_{\ell}^2 \sigma_n^2} } - \frac{\color{group6} (\cancel{\sigma_n^2} x_*) (\sigma_n^2 x_*) }{\sigma_{\ell}^2 \cancel{\sigma_n^2} } - \frac{ 2 {\color{group6} (\cancel{\sigma_n^2} x_*) (\cancel{\sigma_{\ell}^2} \mu_n) } }{\cancel{\sigma_{\ell}^2 \sigma_n^2} } - \frac{\color{group6} (\cancel{\sigma_{\ell}^2} \mu_n) (\sigma_{\ell}^2 \mu_n) }{\cancel{\sigma_{\ell}^2} \sigma_n^2 } \right] \right\} }. $$

From here, it is straightforward to rewrite the expression as a Gaussian:

$$ \frac{ 1 }{\sqrt{2 \pi \sigma_*^2} } \times \exp{ \left\{ \frac{ - 1 }{2 \sigma_*^2 } \left[ {\color{group6} x_*^2} {\color{group2} \cancel{+ \frac{\sigma_{\ell}^2 \mu_n^2}{\sigma_n^2}}} {\color{group5} \cancel{+ \frac{\sigma_n^2 x_*^2}{\sigma_{\ell}^2}}} {\color{group6} + \mu_n^2} {\color{group5} \cancel{- \frac{\sigma_n^2 x_*^2}{\sigma_{\ell}^2}}} {\color{group6} - 2 \mu_n x_*} {\color{group2} \cancel{- \frac{\sigma_{\ell}^2 \mu_n^2}{\sigma_n^2}}} \right] \right\} } \\ = \frac{1}{\sqrt{2 \pi \sigma_*^2}} \times \exp{ \left\{ \frac{- 1}{2 \sigma_*^2} \left[ {\color{group6} x_*^2 - 2 \mu_n x_* + \mu_n^2} \right] \right\} } \\ = \frac{1}{\sqrt{2 \pi {\color{group1} \sigma_*^2}}} \times \exp{ \left\{ \frac{- 1}{2 {\color{group1} \sigma_*^2}} {\color{group6} \left[ x_* - \mu_n \right]^2 } \right\} } \\ = \normalAbbreviation{ \color{group6} x_* }{\color{group6} \mu_n }{\color{group1} \sigma_n^2 + \sigma_{\ell}^2 } $$
Illustrations
Code


Multivariate: Normal-Known

Likelihood of the Training Data
$$ \begin{align} p_{\color{group3} \ell} (\bm{x}_1, \bm{x}_2, \dots, \bm{x}_n \given {\color{group6} \Theta}) &= \prod\limits_i^n \normalAbbreviation{ \bm{x}_i }{\bm{\color{group6} \mu}_{\color{group3} \ell} }{\bm{\color{group6} \Sigma}_{\color{group3} \ell} } \tag{1} \label{multivariateNormalKnown1} \\ &= \frac{1 }{\left(2 \pi \right)^{nD/2} \left\vert \bm{\Sigma}_{\ell} \right\vert^{n/2} } \exp{ \left\{ \frac{-1}{2} \sum\limits_i^n {\color{group1} \left( \bm{x}_i - \bm{\mu}_{\ell} \right)^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group2} \left( \bm{x}_i - \bm{\mu}_{\ell} \right) } \right\} } \tag{2} \label{multivariateNormalKnown2} \\ &= C_1 \times \exp{ \left\{ \frac{-1}{2} \sum\limits_i^n {\color{group1} \left[ (\bm{x}_i - \bar{\bm{x}}) - (\bm{\mu}_{\ell} - \bar{\bm{x}}) \right]^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group2} \left[ (\bm{x}_i - \bar{\bm{x}}) - (\bm{\mu}_{\ell} - \bar{\bm{x}}) \right] } \right\} } \tag{3} \label{multivariateNormalKnown3} \\ &= \circleEquation{ C_1 \times \exp{ \left\{ \frac{-1}{2} tr( \bm{\Sigma}_{\ell}^{-1} \bm{S} ) \right\} } \times {\color{group5} \exp{ \left\{ \frac{-1}{2} (\bar{\bm{x}} - \bm{\mu}_{\ell})^{\top} (n \bm{\Sigma}_{\ell}^{-1}) (\bar{\bm{x}} - \bm{\mu}_{\ell}) \right\} } } } \tag{4} \label{multivariateNormalKnown4} \\ &\propto {\color{group5} \normalAbbreviation{ \bar{\bm{x}} }{\bm{\mu}_{\ell} }{\frac{1}{n} \bm{\Sigma}_{\ell} } } \tag{5} \label{multivariateNormalKnown5} \\ \end{align} $$
Equation 1: Assume the data is conditionally independent under a multivariate Gaussian.

From conditional independence, the (joint) likelihood of data is equal to the product of the individual likelihoods:

$$ \likelihood{ {\color{group1} \bm{x}_1}, {\color{group1} \bm{x}_2}, \dots, {\color{group1} \bm{x}_n} }{\color{group2} \Theta } = \likelihood{\color{group1} \bm{x}_1}{\color{group2} \Theta} \times \likelihood{\color{group1} \bm{x}_2}{\color{group2} \Theta} \times \dots \times \likelihood{\color{group1} \bm{x}_n}{\color{group2} \Theta}. $$

Since we assume the individual likelihoods are Gaussian, :

$$ {\color{group2} p_{\ell}}(\bm{x}_i \given \Theta) \times {\color{group2} p_{\ell}}(\bm{x}_2 \given \Theta) \times \dots \times {\color{group2} p_{\ell}}(\bm{x}_n \given \Theta) \\ = {\color{group2} \mathcal{N}} (\bm{x}_1 \given \bm{\mu}_{\color{group2} \ell}, \bm{\sigma}_{\color{group2} \ell}^2) \times {\color{group2} \mathcal{N}} (\bm{x}_2 \given \bm{\mu}_{\color{group2} \ell}, \bm{\sigma}_{\color{group2} \ell}^2) \times \dots \times {\color{group2} \mathcal{N}} (\bm{x}_n \given \bm{\mu}_{\color{group2} \ell}, \bm{\sigma}_{\color{group2} \ell}^2). $$

Rewrite the product using notation:

$$ \normalAbbreviation{ \color{group5} \bm{x}_1 }{\bm{\mu}_{\ell} }{\bm{\sigma}_{\ell}^2 } {\color{group5} \times} \normalAbbreviation{ \color{group5} \bm{x}_2 }{\bm{\mu}_{\ell} }{\bm{\sigma}_{\ell}^2 } {\color{group5} \times} \dots {\color{group5} \times} \normalAbbreviation{\color{group5} \bm{x}_n }{\bm{\mu}_{\ell} }{\bm{\sigma}_{\ell}^2 } \\ = {\color{group5} \prod\limits_i^n} \normalAbbreviation{ \color{group5} \bm{x}_i }{\bm{\mu}_{\ell} }{\bm{\sigma}_{\ell}^2}. $$
Equation 2: Simplify constants and exponents in the product.

Write out the product by plugging into the Gaussian expression.

$$ \begin{align} \prod\limits_i^n \normalAbbreviation{ \color{group1} \bm{x}_i }{\color{group2} \bm{\mu}_{\ell} }{\color{group2} \bm{\sigma}_{\ell}^2 } &= \frac{1 }{\left(2 \pi \right)^{D/2} \left\vert {\color{group2} \bm{\Sigma}_{\ell}} \right\vert^{1/2} } \exp{ \left\{ \frac{-1}{2} {\left( {\color{group1} \bm{x}_1} - \bm{\mu}_{\ell} \right)^{\top} } {\color{group2} \bm{\Sigma}_{\ell}}^{-1} {\left( {\color{group1} \bm{x}_1} - \bm{\mu}_{\ell} \right) } \right\} } \\ &\times \frac{1 }{\left(2 \pi \right)^{D/2} \left\vert {\color{group2} \bm{\Sigma}_{\ell}} \right\vert^{1/2} } \exp{ \left\{ \frac{-1}{2} {\left( {\color{group1} \bm{x}_2} - \bm{\mu}_{\ell} \right)^{\top} } {\color{group2} \bm{\Sigma}_{\ell}}^{-1} {\left( {\color{group1} \bm{x}_2} - \bm{\mu}_{\ell} \right) } \right\} } \\ &\times \dots \\ &\times \frac{1 }{\left(2 \pi \right)^{D/2} \left\vert {\color{group2} \bm{\Sigma}_{\ell}} \right\vert^{1/2} } \exp{ \left\{ \frac{-1}{2} {\left( {\color{group1} \bm{x}_n} - \bm{\mu}_{\ell} \right)^{\top} } {\color{group2} \bm{\Sigma}_{\ell}}^{-1} {\left( {\color{group1} \bm{x}_n} - \bm{\mu}_{\ell} \right) } \right\} } \end{align} $$

Bring the normalizing constants to the front.

$$ \begin{align} {\color{group5} \frac{1 }{\left(2 \pi \right)^{D/2} \left\vert \bm{\Sigma}_{\ell} \right\vert^{1/2} } } \exp{ \left\{ \frac{-1}{2} {\left( \bm{x}_1 - \bm{\mu}_{\ell} \right)^{\top} } \bm{\Sigma}_{\ell}^{-1} {\left( \bm{x}_1 - \bm{\mu}_{\ell} \right) } \right\} } \\ \times {\color{group5} \frac{1 }{\left(2 \pi \right)^{D/2} \left\vert \bm{\Sigma}_{\ell} \right\vert^{1/2} } } \exp{ \left\{ \frac{-1}{2} {\left( \bm{x}_2 - \bm{\mu}_{\ell} \right)^{\top} } {\bm{\Sigma}_{\ell}}^{-1} {\left( \bm{x}_2 - \bm{\mu}_{\ell} \right) } \right\} } \\ \times \dots \times {\color{group5} \frac{1 }{\left(2 \pi \right)^{D/2} \left\vert \bm{\Sigma}_{\ell} \right\vert^{1/2} } } \exp{ \left\{ \frac{-1}{2} {\left( \bm{x}_n - \bm{\mu}_{\ell} \right)^{\top} } {\bm{\Sigma}_{\ell}}^{-1} {\left( \bm{x}_n - \bm{\mu}_{\ell} \right) } \right\} } \\ = {\color{group5} \left( \frac{1 }{\left(2 \pi \right)^{D/2} \left\vert \bm{\Sigma}_{\ell} \right\vert^{1/2} } \right)^n } \times \exp{ \left\{ \frac{-1}{2} {\left( \bm{x}_1 - \bm{\mu}_{\ell} \right)^{\top} } \bm{\Sigma}_{\ell}^{-1} {\left( \bm{x}_1 - \bm{\mu}_{\ell} \right) } \right\} } \\ \times \exp{ \left\{ \frac{-1}{2} {\left( \bm{x}_2 - \bm{\mu}_{\ell} \right)^{\top} } {\bm{\Sigma}_{\ell}}^{-1} {\left( \bm{x}_2 - \bm{\mu}_{\ell} \right) } \right\} } \\ \times \dots \times \exp{ \left\{ \frac{-1}{2} {\left( \bm{x}_n - \bm{\mu}_{\ell} \right)^{\top} } {\bm{\Sigma}_{\ell}}^{-1} {\left( \bm{x}_n - \bm{\mu}_{\ell} \right) } \right\} } \\ \end{align} $$

Sum exponents. Recall that .

$$ \left( \frac{1 }{\left(2 \pi \right)^{D/2} \left\vert \bm{\Sigma}_{\ell} \right\vert^{1/2} } \right)^n \\ \times \exp{ \left\{ \frac{-1}{2} {\left( \bm{x}_1 - \bm{\mu}_{\ell} \right)^{\top} } \bm{\Sigma}_{\ell}^{-1} {\left( \bm{x}_1 - \bm{\mu}_{\ell} \right) } \right\} } \\ {\color{group5} \times} \exp{ \left\{ \frac{-1}{2} {\left( \bm{x}_2 - \bm{\mu}_{\ell} \right)^{\top} } {\bm{\Sigma}_{\ell}}^{-1} {\left( \bm{x}_2 - \bm{\mu}_{\ell} \right) } \right\} } \\ {\color{group5} \times} \dots {\color{group5} \times} \exp{ \left\{ \frac{-1}{2} {\left( \bm{x}_n - \bm{\mu}_{\ell} \right)^{\top} } {\bm{\Sigma}_{\ell}}^{-1} {\left( \bm{x}_n - \bm{\mu}_{\ell} \right) } \right\} } \\ = \left( \frac{1 }{\left(2 \pi \right)^{D/2} \left\vert \bm{\Sigma}_{\ell} \right\vert^{1/2} } \right)^n \times \exp{ \left\{ {\color{group5} \sum\limits_i^n } \frac{-1}{2} {\left( \bm{x}_i - \bm{\mu}_{\ell} \right)^{\top} } {\bm{\Sigma}_{\ell}}^{-1} {\left( \bm{x}_i - \bm{\mu}_{\ell} \right) } \right\} } $$

Distribute in the normalizing constant and factor in the exponent.

$$ \begin{align} \left( \frac{1 }{\left(2 \pi \right)^{D/2} \left\vert \bm{\Sigma}_{\ell} \right\vert^{1/2} } \right)^{\color{group5} n} \times \exp{ \left\{ \sum\limits_i^n {\color{group6} \frac{-1}{2}} {\left( \bm{x}_i - \bm{\mu}_{\ell} \right)^{\top} } {\bm{\Sigma}_{\ell}}^{-1} {\left( \bm{x}_i - \bm{\mu}_{\ell} \right) } \right\} } \\ = \frac{1 }{\left(2 \pi \right)^{\color{group5} n D/2} \left\vert \bm{\Sigma}_{\ell} \right\vert^{\color{group5} n /2} } \times \exp{ \left\{ {\color{group6} \frac{-1}{2}} \sum\limits_i^n {\left( \bm{x}_i - \bm{\mu}_{\ell} \right)^{\top} } {\bm{\Sigma}_{\ell}}^{-1} {\left( \bm{x}_i - \bm{\mu}_{\ell} \right) } \right\} } \end{align} $$
Equation 3: Introduce the empirical mean via .
$$ \begin{align} \frac{1 }{\left(2 \pi \right)^{n D/2} \left\vert \bm{\Sigma}_{\ell} \right\vert^{n /2} } \times \exp{ \left\{ \frac{-1}{2} \sum\limits_i^n {\left( \bm{x}_i - \bm{\mu}_{\ell} \right)^{\top} } {\bm{\Sigma}_{\ell}}^{-1} {\left( \bm{x}_i - \bm{\mu}_{\ell} \right) } \right\} } \\ = \frac{1 }{\left(2 \pi \right)^{n D/2} \left\vert \bm{\Sigma}_{\ell} \right\vert^{n /2} } \times \exp{ \left\{ \frac{-1}{2} \sum\limits_i^n \left( \bm{x}_i + {\color{group5} 0} - \bm{\mu}_{\ell} \right)^{\top} {\bm{\Sigma}_{\ell}}^{-1} \left( \bm{x}_i + {\color{group5} 0} - \bm{\mu}_{\ell} \right) \right\} } \\ = \frac{1 }{\left(2 \pi \right)^{n D/2} \left\vert \bm{\Sigma}_{\ell} \right\vert^{n /2} } \times \exp{ \left\{ \frac{-1}{2} \sum\limits_i^n \left( \bm{x}_i + {\color{group5} \bar{\bm{x}} - \bar{\bm{x}}} - \bm{\mu}_{\ell} \right)^{\top} {\bm{\Sigma}_{\ell}}^{-1} \left( \bm{x}_i + {\color{group5} \bar{\bm{x}} - \bar{\bm{x}}} - \bm{\mu}_{\ell} \right) \right\} } \\ = \frac{1 }{\left(2 \pi \right)^{n D/2} \left\vert \bm{\Sigma}_{\ell} \right\vert^{n /2} } \times \exp{ \left\{ \frac{-1}{2} \sum\limits_i^n \left[ (\bm{x}_i - {\color{group5} \bar{\bm{x}}}) {\color{group6} +} ( {\color{group5} \bar{\bm{x}}} {\color{group6} -} {\color{group1} \bm{\mu}_{\ell}} ) \right]^{\top} {\bm{\Sigma}_{\ell}}^{-1} \left[ (\bm{x}_i - {\color{group5} \bar{\bm{x}}}) {\color{group6} +} ( {\color{group5} \bar{\bm{x}}} {\color{group6} -} {\color{group1} \bm{\mu}_{\ell}} ) \right] \right\} } \\ = \frac{1 }{\left(2 \pi \right)^{n D/2} \left\vert \bm{\Sigma}_{\ell} \right\vert^{n /2} } \times \exp{ \left\{ \frac{-1}{2} \sum\limits_i^n \left[ (\bm{x}_i - {\color{group5} \bar{\bm{x}}}) {\color{group6} -} ( {\color{group1} \bm{\mu}_{\ell}} {\color{group6} -} {\color{group5} \bar{\bm{x}}} ) \right]^{\top} {\bm{\Sigma}_{\ell}}^{-1} \left[ (\bm{x}_i - {\color{group5} \bar{\bm{x}}}) {\color{group6} -} ( {\color{group1} \bm{\mu}_{\ell}} {\color{group6} -} {\color{group5} \bar{\bm{x}}} ) \right] \right\} } \\ \end{align} $$

Letting

$$ {\color{group2} \frac{1 }{\left(2 \pi \right)^{n D/2} \left\vert \bm{\Sigma}_{\ell} \right\vert^{n /2} } } \times \exp{ \left\{ \frac{-1}{2} \sum\limits_i^n \left[ (\bm{x}_i - {\color{group5} \bar{\bm{x}}}) {\color{group6} -} ( {\color{group1} \bm{\mu}_{\ell}} {\color{group6} -} {\color{group5} \bar{\bm{x}}} ) \right]^{\top} {\bm{\Sigma}_{\ell}}^{-1} \left[ (\bm{x}_i - {\color{group5} \bar{\bm{x}}}) {\color{group6} -} ( {\color{group1} \bm{\mu}_{\ell}} {\color{group6} -} {\color{group5} \bar{\bm{x}}} ) \right] \right\} } \\ = {\color{group2} C_1} \times \exp{ \left\{ \frac{-1}{2} \sum\limits_i^n \left[ (\bm{x}_i - {\color{group5} \bar{\bm{x}}}) {\color{group6} -} ( {\color{group1} \bm{\mu}_{\ell}} {\color{group6} -} {\color{group5} \bar{\bm{x}}} ) \right]^{\top} {\bm{\Sigma}_{\ell}}^{-1} \left[ (\bm{x}_i - {\color{group5} \bar{\bm{x}}}) {\color{group6} -} ( {\color{group1} \bm{\mu}_{\ell}} {\color{group6} -} {\color{group5} \bar{\bm{x}}} ) \right] \right\} } \\ $$
Equation 4: Expand the quadratic expression, and then simplify, letting represent the scatter matrix.

We expand by first FOILing the quadratic expression in the summation of the exponent:

$$ \begin{align} \left[ {\color{group5} (\bm{x}_i - \bar{\bm{x}}) } - {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } \right]^{\top} \bm{\Sigma}_{\ell}^{-1} \left[ {\color{group5} (\bm{x}_i - \bar{\bm{x}}) } - {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } \right] &={\color{group5} (\bm{x}_i - \bar{\bm{x}})^{\top} } \bm{\Sigma}_{\ell}^{-1} \left[ {\color{group5} (\bm{x}_i - \bar{\bm{x}}) } - {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } \right] - {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}})^{\top} } \bm{\Sigma}_{\ell}^{-1} \left[ {\color{group5} (\bm{x}_i - \bar{\bm{x}}) } - {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } \right] \\ &={\color{group5} (\bm{x}_i - \bar{\bm{x}})^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group5} (\bm{x}_i - \bar{\bm{x}}) } - {\color{group5} (\bm{x}_i - \bar{\bm{x}})^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } - {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}})^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group5} (\bm{x}_i - \bar{\bm{x}}) } + {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}})^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } \end{align} $$

Since a scalar is equal to the transpose of itself, , and a symmetric matrix is equal to the transpose of itself, , we can simplify the two middle terms in the expression above::

$$ - {\color{group5} (\bm{x}_i - \bar{\bm{x}})^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } - {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}})^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group5} (\bm{x}_i - \bar{\bm{x}}) } \\ = - {\color{group5} (\bm{x}_i - \bar{\bm{x}})^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } - \left[ {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}})^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group5} (\bm{x}_i - \bar{\bm{x}}) } \right]^{\top} \\ = - {\color{group5} (\bm{x}_i - \bar{\bm{x}})^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } - \left[ {\color{group5} (\bm{x}_i - \bar{\bm{x}}) }^{\top} \bm{\Sigma}_{\ell}^{- \top} {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}})^{\top} }^{\top} \right] \\ = - {\color{group5} (\bm{x}_i - \bar{\bm{x}})^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } - {\color{group5} (\bm{x}_i - \bar{\bm{x}}) }^{\top} \bm{\Sigma}_{\ell}^{-1} {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } \\ = - 2 {\color{group5} (\bm{x}_i - \bar{\bm{x}})^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } $$

Now, reintroducing and distributing the summation in the exponent,

$$ \sum\limits_i^n \left[ (\bm{x}_i - \bar{\bm{x}})^{\top} \bm{\Sigma}_{\ell}^{-1} (\bm{x}_i - \bar{\bm{x}}) - 2 {\color{group5} (\bm{x}_i - \bar{\bm{x}})^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } + (\bm{\mu}_{\ell} - \bar{\bm{x}})^{\top} \bm{\Sigma}_{\ell}^{-1} (\bm{\mu}_{\ell} - \bar{\bm{x}}) \right] \\ = \left[ \sum\limits_i^n (\bm{x}_i - \bar{\bm{x}})^{\top} \bm{\Sigma}_{\ell}^{-1} (\bm{x}_i - \bar{\bm{x}}) \right] + \left[ \sum\limits_i^n - 2 {\color{group5} (\bm{x}_i - \bar{\bm{x}})^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } \right] + \left[ \sum\limits_i^n (\bm{\mu}_{\ell} - \bar{\bm{x}})^{\top} \bm{\Sigma}_{\ell}^{-1} (\bm{\mu}_{\ell} - \bar{\bm{x}}) \right]. $$

Recalling that , we show that the middle set of terms sum to :

$$ \sum\limits_i^n {\color{group3} - 2} {\color{group5} (\bm{x}_i - \bar{\bm{x}})^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } = {\color{group3} - 2} \sum\limits_i^n {\color{group5} (\bm{x}_i - \bar{\bm{x}})^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } \\ = - 2 \left[ \sum\limits_i^n {\color{group5} \bm{x}_i}^{\top} \bm{\Sigma}_{\ell}^{-1} {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } \ {\color{group5} -} \ {\color{group5} \bar{\bm{x}}^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } \right] \\ = - 2 \left[ n {\color{group5} \bar{\bm{x}}}^{\top} \bm{\Sigma}_{\ell}^{-1} {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } \ {\color{group5} -} \ n {\color{group5} \bar{\bm{x}}^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } \right] \\ = -2 \times 0 = 0. $$

This means

$$ \left[ \sum\limits_i^n (\bm{x}_i - \bar{\bm{x}})^{\top} \bm{\Sigma}_{\ell}^{-1} (\bm{x}_i - \bar{\bm{x}}) \right] + \left[ \sum\limits_i^n - 2 {\color{group5} (\bm{x}_i - \bar{\bm{x}})^{\top} } \bm{\Sigma}_{\ell}^{-1} {\color{group6} (\bm{\mu}_{\ell} - \bar{\bm{x}}) } \right] + \left[ \sum\limits_i^n (\bm{\mu}_{\ell} - \bar{\bm{x}})^{\top} \bm{\Sigma}_{\ell}^{-1} (\bm{\mu}_{\ell} - \bar{\bm{x}}) \right] \\ = \left[ \sum\limits_i^n (\bm{x}_i - \bar{\bm{x}})^{\top} \bm{\Sigma}_{\ell}^{-1} (\bm{x}_i - \bar{\bm{x}}) \right] + \left[ \sum\limits_i^n (\bm{\mu}_{\ell} - \bar{\bm{x}})^{\top} \bm{\Sigma}_{\ell}^{-1} (\bm{\mu}_{\ell} - \bar{\bm{x}}) \right]. $$

Since is constant with respect to the indices , and are allowed to commute constants, we have

$$ \left[ \sum\limits_i^n (\bm{x}_i - \bar{\bm{x}})^{\top} \bm{\Sigma}_{\ell}^{-1} (\bm{x}_i - \bar{\bm{x}}) \right] + \left[ {\color{group3} \sum\limits_i^n} (\bm{\mu}_{\ell} - \bar{\bm{x}})^{\top} \bm{\Sigma}_{\ell}^{-1} (\bm{\mu}_{\ell} - \bar{\bm{x}}) \right] \\ = \left[ \sum\limits_i^n (\bm{x}_i - \bar{\bm{x}})^{\top} \bm{\Sigma}_{\ell}^{-1} (\bm{x}_i - \bar{\bm{x}}) \right] + {\color{group3} n} (\bm{\mu}_{\ell} - \bar{\bm{x}})^{\top} \bm{\Sigma}_{\ell}^{-1} (\bm{\mu}_{\ell} - \bar{\bm{x}}) \\ = \left[ \sum\limits_i^n (\bm{x}_i - \bar{\bm{x}})^{\top} \bm{\Sigma}_{\ell}^{-1} (\bm{x}_i - \bar{\bm{x}}) \right] + (\bm{\mu}_{\ell} - \bar{\bm{x}})^{\top} \left( {\color{group3} n} \bm{\Sigma}_{\ell}^{-1} \right) (\bm{\mu}_{\ell} - \bar{\bm{x}}). $$

So, plugging back in, we have the following likelihood, where the notation for the remaining summation can be further simplified:

$$ \frac{1 }{\left(2 \pi \right)^{nD/2} \left\vert \bm{\Sigma} \right\vert^{n/2} } \exp{ \left\{ \frac{-1}{2} {\color{teal} \left[ \sum\limits_i^n (\bm{x}_i - \bar{\bm{x}})^{\top} \bm{\Sigma}^{-1} (\bm{x}_i - \bar{\bm{x}}) \right] } - \frac{1}{2} \left[ (\bm{\mu}_{\ell} - \bar{\bm{x}})^{\top} (n \bm{\Sigma}_{\ell}^{-1}) (\bm{\mu}_{\ell} - \bar{\bm{x}}) \right] \right\} } $$

Since the trace of a scalar is equal to itself, , and the result of the remaining summation is a scalar, we can simplify the notation with a little work:

$$ {\color{teal} \sum\limits_i^n (\bm{x}_i - \bar{\bm{x}})^{\top} \bm{\Sigma}^{-1} (\bm{x}_i - \bar{\bm{x}}) } = tr \left( {\color{teal} \sum\limits_i^n (\bm{x}_i - \bar{\bm{x}})^{\top} \bm{\Sigma}^{-1} (\bm{x}_i - \bar{\bm{x}}) } \right). $$

Since we know that given two vectors and ,

$$ \begin{align} tr \left( \sum\limits_i^n {\color{group1} (\bm{x}_i - \bar{\bm{x}})^{\top} } {\color{group2} \bm{\Sigma}^{-1} (\bm{x}_i - \bar{\bm{x}}) } \right) &= tr \left( \sum\limits_i^n {\color{group2} \bm{\Sigma}^{-1} (\bm{x}_i - \bar{\bm{x}}) } {\color{group1} (\bm{x}_i - \bar{\bm{x}})^{\top} } \right). \end{align} $$

Factoring ,

$$ tr \left( \sum\limits_i^n {\color{group5} \bm{\Sigma}^{-1}} (\bm{x}_i - \bar{\bm{x}}) (\bm{x}_i - \bar{\bm{x}})^{\top} \right) = tr \left( {\color{group5} \bm{\Sigma}^{-1}} \sum\limits_i^n (\bm{x}_i - \bar{\bm{x}}) (\bm{x}_i - \bar{\bm{x}})^{\top} \right). $$

Recognizing as a scatter matrix, we can simplify our notation:

$$ tr \left( {\color{group5} \bm{\Sigma}^{-1}} {\color{group6} \sum\limits_i^n (\bm{x}_i - \bar{\bm{x}}) (\bm{x}_i - \bar{\bm{x}})^{\top} } \right) = tr \left( {\color{group5} \bm{\Sigma}^{-1}} {\color{group6} \bm{S}} \right). $$

Plugging back in, we have the following:

$$ \frac{1 }{\left(2 \pi \right)^{nD/2} \left\vert \bm{\Sigma} \right\vert^{n/2} } \times \exp{ \left\{ \frac{-1}{2} {\color{teal} tr \left( \bm{\Sigma}^{-1} \bm{S} \right) } - \frac{1}{2} \left[ (\bm{\mu}_{\ell} - \bar{\bm{x}})^{\top} (n \bm{\Sigma}^{-1}) (\bm{\mu}_{\ell} - \bar{\bm{x}}) \right] \right\} } \\ = \frac{1 }{\left(2 \pi \right)^{nD/2} \left\vert \bm{\Sigma} \right\vert^{n/2} } \times \exp{ \left\{ \frac{-1}{2} {\color{teal} tr \left( \bm{\Sigma}^{-1} \bm{S} \right) } \right\} } \times \exp{ \left\{ - \frac{1}{2} \left[ (\bm{\mu}_{\ell} - \bar{\bm{x}})^{\top} (n \bm{\Sigma}^{-1}) (\bm{\mu}_{\ell} - \bar{\bm{x}}) \right] \right\} } $$
Equation 5: Note the proportionality to the sampling distribution of the empirical mean.

Recall that , meaning that it does not depend on the likelihood mean . Since the only variable in the Normal-Known model is , this makes the second factor constant.

Dropping and , we see that the remaining the exponential factor is proportional to a Gaussian with mean and covariance .

$$ {\color{group6} \frac{1 }{\left(2 \pi \right)^{nD/2} \left\vert \bm{\Sigma}_{\ell} \right\vert^{n/2} } \times \exp{ \left\{ \frac{-1}{2} tr \left( \bm{\Sigma}_{\ell}^{-1} \bm{S} \right) \right\} } \times } \exp{ \left\{ - \frac{1}{2} \left[ (\bm{\mu}_{\ell} - \bar{\bm{x}})^{\top} (n \bm{\Sigma}_{\ell}^{-1}) (\bm{\mu}_{\ell} - \bar{\bm{x}}) \right] \right\} } \\ {\color{group6} \propto} \ \exp{ \left\{ - \frac{1}{2} \left[ (\bm{\mu}_{\ell} - \bar{\bm{x}})^{\top} (n \bm{\Sigma}_{\ell}^{-1}) (\bm{\mu}_{\ell} - \bar{\bm{x}}) \right] \right\} } \\ {\color{group6} \propto} \ \normalAbbreviation{ \bar{\bm{x}} }{\bm{\mu_{\ell}} }{\bm{\Sigma}_{\ell} } $$
Prior on the True Mean
Equation 1:
Equation 2:
Marginal Likelihood of the Training Data
Equation 1:
Equation 2:
Posterior on the True Mean and Variance
Equation 1:
Equation 2:
Posterior Predictive of Test Data
Equation 1:
Equation 2:
Illustrations
Code
Code