View the assignment description here

Exercise 1

$\newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}}$ The MSE (= Mean Squared Error) of the estimator $\hat{\vect\theta}$ is $$MSE\left(\hat{\vect\theta}\right) = \mathbb E _{p\left( \mathcal D | \vect \theta_0 \right)} \left[ \left(\hat{\vect\theta} - \vect \theta_0\right)^2 \right]$$ where $\vect \theta_0$ is the true parameter.

We can rewrite this by subtracting and adding the squared bias term $bias\left(\hat{\vect\theta}\right)^2= \left(\mathbb E _{p\left( \mathcal D | \vect \theta_0 \right)} \left[\hat{\vect \theta}\right] - \vect \theta_0\right)^2 =\mathbb E _{p\left( \mathcal D | \vect \theta_0 \right)} \left[\hat{\vect \theta} - \vect \theta_0\right]^2$ $$MSE\left(\hat{\vect\theta}\right) = \mathbb E _{p\left( \mathcal D | \vect \theta_0 \right)} \left[ \left(\hat{\vect\theta} - \vect \theta_0\right)^2 \right]

  • \mathbb E _{p\left( \mathcal D | \vect \theta_0 \right)} \left[
      \hat{\vect \theta} - \vect \theta_0\right]^2
  • \mathbb E _{p\left( \mathcal D | \vect \theta_0 \right)} \left[
      \hat{\vect \theta} - \vect \theta_0\right]^2$$

And now since $Var\left[ X \right] = \mathbb E\left[ X^2 \right] - \mathbb E\left[ X \right]^2$ $$MSE\left(\hat{\vect\theta}\right) = Var\left[\hat{\vect\theta} - \vect\theta_0\right] + bias\left(\hat{\vect\theta}\right)^2$$

finally since $Var\left[ X + c \right] = Var\left[X \right]$ $$MSE\left(\hat{\vect\theta}\right) = Var\left[\hat{\vect\theta}\right] + bias\left(\hat{\vect\theta}\right)^2$$