View the assignment description here

$\newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}}$ Let $\vect x \in \mathbb R^n$ and consider the multivariate Gaussian distribution $$p\left( \vect x | \vect \mu \vect, \vect \Sigma \right) = | 2 \pi \vect \Sigma |^{-1/2} \exp \left\{ -\frac{1}{2} \left( \vect x - \vect \mu \right)^T \vect \Sigma^{-1} \left(\vect x - \vect \mu \right) \right\}$$

The differential entropy of this distribution is given by $$H\left[ p \right] =

  • \int p\left(\vect x\right) \log p\left( \vect x \right)\mathrm d\vect x$$ $$=- \int p\left(\vect x\right) \log \left( | 2 \pi \vect \Sigma |^{-1/2} \exp \left{ -\frac{1}{2} \left( \vect x - \vect \mu \right)^T \vect \Sigma^{-1} \left(\vect x - \vect \mu \right) \right} \right)\mathrm d\vect x$$ $$ = \frac{1}{2} \log |2 \pi \vect \Sigma | \int p\left(\vect x \right) \mathrm d \vect x
  • \frac{1}{2} \int p\left(\vect x \right) \left( \vect x - \vect \mu \right)^T \vect \Sigma^{-1} \left(\vect x - \vect \mu \right) \mathrm d \vect x$$ $$ = \frac{1}{2} \log |2 \pi \vect \Sigma |
  • \frac{1}{2} \mathbb E \left[ \left( \vect x - \vect \mu \right)^T \vect \Sigma^{-1} \left(\vect x - \vect \mu \right) \right]$$ $$ = \frac{1}{2} \log |2 \pi \vect \Sigma |
  • \frac{1}{2} \mathbb E \left[ \sum{i=1}^n \sum{j=1}^n \vect \Sigma_{ij}^{-1} \left(\vect x - \vect \mu\right)_j \left(\vect x - \vect \mu\right)_i \right]$$ $$ = \frac{1}{2} \log |2 \pi \vect \Sigma |
  • \frac{1}{2} \sum{i=1}^n \sum{j=1}^n \mathbb E \left[ \left(\vect x - \vect \mu\right)_j \left(\vect x - \vect \mu\right)i \right] \vect \Sigma{ij}^{-1}$$ $$ = \frac{1}{2} \log |2 \pi \vect \Sigma |
  • \frac{1}{2} \sum{j=1}^n \sum{i=1}^n \vect \Sigma{ji} \vect \Sigma{ij}^{-1} $$ $$ = \frac{1}{2} \log |2 \pi \vect \Sigma |
  • \frac{1}{2} \sum{j=1}^n \left( \vect \Sigma \vect \Sigma^{-1}\right){jj} $$ $$ = \frac{1}{2} \log |2 \pi \vect \Sigma |
  • \frac{1}{2} n $$ $$ = \frac{1}{2} \log |2 \pi \vect \Sigma |
  • \frac{1}{2} \log e^n = \frac{1}{2} \log |2 \pi e \Sigma| $$