(Defining latex commands: not to be shown...) $$ \newcommand{\norm}[1]{\left \| #1 \right \|} \DeclareMathOperator{\minimize}{minimize} \newcommand{\real}{\mathbb{R}} \newcommand{\normal}{\mathcal{N}} $$
Prove that the product of two univariate (scalar) Gaussian distributions is a Gaussian again, i.e. show, by explicitly performing the required arithmetic transformations, that
\begin{equation} \normal(x;\mu,\sigma^2)\normal(x;m,s^2) = \normal[x; (\frac 1{\sigma^2}+\frac 1{s^2})^{-1}(\frac \mu{\sigma^2}+\frac m{s^2}),(\frac 1{\sigma^2}+\frac 1{s^2})^{-1}]\normal[m,\mu,\sigma^2+s^2]. \end{equation}Derive the formula $\mathbf{w}_{MLE} = (X^TX)^{-1}X^T\mathbf{y}$ from the lecture, by calculating the derivative of $p(\mathbf{y}\,|X,\mathbf{w}) = \normal(\mathbf{y}\,|X\mathbf{w}, \sigma^2I)$ with respect to $\mathbf{w}$, setting it to zero and solving it for $\mathbf{w}$.
Note: To refresh your linear algebra you might find it useful to have a look in here.
In this exercise you will perform a regression analysis on a toy dataset. You will implement ridge regression and learn how to find a good model through a comparative performance analysis.
# Loading the required packages
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from IPython.html.widgets import interact
def wRidge(X,y,lamb):
# Change the following line and implement the ridge regression estimator wRidge
w = np.zeros(X.shape[-1])
return w
# Read ex1_train.csv and assign the first column and
# second column to variables x and y respectively.
# Plot the input data here
Note: Constructing higher polynomial features is similar to what you implemented in Exercise 3 (SVM) of the previous exercise sheet.
def construct_poly(x,d):
## Implement a method which given an array of size N,
## returns an array of dimension (N,d)
return np.zeros((x.shape[0],d+x.shape[1]))
def MSE(y_predict,y_true):
## Implement mean squared error for a given input y and its predictions.
return 0.0
##This function provides an interactive mode to change polynomial degree.
@interact(n=[1,16])
def plot(n):
X = construct_poly(x,n)
w = wRidge(X,y,1.0)
plt.plot(x,X.dot(w))
plt.title("MSE %f" % MSE(X.dot(w),y))
plt.plot(x,y)
## Read test data here
Hand in printed copy of completed notebook.