#!/usr/bin/env python
# coding: utf-8

# (Defining latex commands: not to be shown...)
# $$
# \newcommand{\norm}[1]{\left \| #1 \right \|}
# \DeclareMathOperator{\minimize}{minimize}
# \newcommand{\real}{\mathbb{R}}
# \newcommand{\normal}{\mathcal{N}}
# $$

# #  Gaussian Algebra (25 Points)
# 
# Prove that the product of two univariate (scalar) Gaussian distributions is a Gaussian again, i.e. show, by explicitly performing the required arithmetic transformations, that
# 
# \begin{equation}
#    \normal(x;\mu,\sigma^2)\normal(x;m,s^2) = \normal[x; (\frac 1{\sigma^2}+\frac 1{s^2})^{-1}(\frac \mu{\sigma^2}+\frac m{s^2}),(\frac 1{\sigma^2}+\frac 1{s^2})^{-1}]\normal[m,\mu,\sigma^2+s^2].
# \end{equation}

# #  Maximum Likelihood Estimator of Simple Linear Regression (25 Points)
# 
# Derive the formula $\mathbf{w}_{MLE} = (X^TX)^{-1}X^T\mathbf{y}$ from the lecture, by calculating the derivative of $p(\mathbf{y}\,|X,\mathbf{w}) = \normal(\mathbf{y}\,|X\mathbf{w}, \sigma^2I)$ with respect to $\mathbf{w}$, setting it to zero and solving it for $\mathbf{w}$.
# 
# 
# Note: _To refresh your linear algebra you might find it useful to have a look in [here](http://webdav.tuebingen.mpg.de/lectures/ei-SS2015/pdfs/Murray_cribsheet.pdf)._

# #  Linear regression (50 Points)
# 
# In this exercise you will perform a regression analysis on a toy dataset. You will implement ridge regression and learn how to find a good model through a comparative performance analysis.
# 
# 1) Download the [training set](http://webdav.tuebingen.mpg.de/lectures/ei-SS2015/data/ex1_train.csv)! <br>
# 2) Implement $\mathbf{w}_{RIDGE}$ as a function of a given $X, \mathbf{y}$ array and a regularization parameter $\lambda$!

# In[ ]:


# Loading the required packages
get_ipython().run_line_magic('matplotlib', 'inline')
import numpy as np
import matplotlib.pyplot as plt
from IPython.html.widgets import interact


# In[ ]:


def wRidge(X,y,lamb):
  # Change the following line and implement the ridge regression estimator wRidge
  w = np.zeros(X.shape[-1])
  return w


# 3) Load "ex1_train.csv" into a numpy array! The first column in the csv file is $X$ and the second column is $\mathbf{y}$, assign them to each variable!

# In[ ]:


# Read ex1_train.csv and assign the first column and 
# second column to variables x and y respectively.


# 4) Plot the training data with appropriate labels on each axes!

# In[ ]:


# Plot the input data here


# 5) Implement a function which constructs features upto a input polynomial degree $d$!<br>
# Note: _Constructing higher polynomial features is similar to what you implemented in Exercise 3 (SVM) of the previous exercise sheet._

# In[68]:


def construct_poly(x,d):
  ## Implement a method which given an array of size N, 
  ## returns an array of dimension (N,d)
  return np.zeros((x.shape[0],d+x.shape[1]))


# 6) Implement the Mean Squared Error Loss (MSE) as a function of the predicted and true values of the target variable! <br>

# In[ ]:


def MSE(y_predict,y_true):
  ## Implement mean squared error for a given input y and its predictions.
  return 0.0


# 7) By comparing the MSE find the degree $d$ for the polynomial that fits the training data best! You might find it useful to use the code below to interactively change the variable $d$, set $\lambda = 1$ and keep it fixed. Plot the error as a function of different values of $d$!<br>

# In[ ]:


##This function provides an interactive mode to change polynomial degree. 
@interact(n=[1,16])
def plot(n):
  X = construct_poly(x,n)
  w = wRidge(X,y,1.0)
  plt.plot(x,X.dot(w))
  plt.title("MSE %f" % MSE(X.dot(w),y))
  plt.plot(x,y)


# 8) Apply models with different values of $d$ after being trained on the training dataset, to the test data available [here](http://webdav.tuebingen.mpg.de/lectures/ei-SS2015/data/ex1_train.csv). Compare the errors on the test data to the ones from the training by plotting the error curves as functions of the polynomial degree in a single plot! What do you conclude? <br>

# In[67]:


## Read test data here


# 9) With a fixed optimal $d$, change the value of $\lambda$ to one of the following values $[0.1, 1.0, 10.0]$ and find the minimum MSE!<br>

# Hand in printed copy of completed notebook.