#!/usr/bin/env python
# coding: utf-8

# # PySeison - Tutorial 6: Validation class

# In[1]:


get_ipython().run_line_magic('pylab', 'inline')


# ## 1. PySeidon-dvt - Validation object initialisation

# "Validation class" is a hybrid class which requires both a numerical-model-based object (e.g. "FVCOM" or "Station" object) and at least one measurement-based object (e.g. "ADCP", "TideGauge" or "Drifter" object or a list of their combination). Its aim is to provide validation benchmarks in order to access the accuracy of a FVCOM simulation compared to matching measurements.

# ### 1.1. Package importation

# As any other library in *Python*, PySeidon-dvt has to be first imported before to be used. Here we will use an alternative *import* statement compared to the one previoulsy presented:  

# In[2]:


from pyseidon_dvt import *


# *** Star *** here means *all*. Usually this form of statements would import the entire library. In the case of *PySeidon-dvt*, this statement will import the following object classes: FVCOM, Station, Validation, ADCP, Tidegauge and Drifter. Only the Validation class will be tackle in this tutorial.

# ### 1.2. Object definition

# *Python* is by definition an [**object oriented language**](http://www.tutorialspoint.com/python/python_classes_objects.htm)...and so is *matlab*. *PySeidon-dvt* is based on this notion of object, so let us define our first "Validation" object.

# ***Exercise 1: ***
# - Unravel Drifter documentation with Ipython ***shortcuts***

# ***Answer: ***

# In[3]:


get_ipython().run_line_magic('pinfo', 'Validation')


# According to the documentation, in order to define a Validation object, the required inputs are:
# - standalone or list of PySeidon-dvt measurement objects (e.g. ADCP, TideGauge, Drifter)
# - standalone or list PySeidon-dvt simulation objects (i.e. FVCOM or Station)
# 
# **Note**:
# - ***flow*** option permits to impose flow comparison by surface flow ('sf'), depth-averaged flow ('daf') or at any depth (float). If this option is not provided, the option choice will be prompted to the user.
# - if the time period between simulation and measurement objects does not match, one can use ***harmonic_reconstruction*** option in order to extrapolate and interpolate measurements' periods based on an harmonic analysis. ***Warning:*** this option may lead to spurious extrapolations.

# ***Exercise 2: ***
# - Initialize adcp, tidegauge and drifter objects from the templates provided in **./data4tutorial/drifter_GP_01aug2013.mat**
# - ***Tip:*** adapt the file's path to your local machine.
# - Initialize station and fvcom objects from the following opendap urls: http://ecoii.acadiau.ca/thredds/dodsC/ecoii/test/Station3D_dngrid_BF_20130730_20130809.nc and http://ecoii.acadiau.ca/thredds/dodsC/ecoii/test/FVCOM3D_dngrid_GP_20130801_20130802.nc
# - Initialize ***val1***, a Validation object using as input the adcp object and the station object and 'flow' option to 10.0 m depth from sea bottom upwards
# - Initialize ***val2***, a Validation object using as input a list of all the measurement objects and the FVCOM object and 'flow' option to 'daf'.
# - Initialize ***val2***, a Validation object using as input a list of all the measurement objects and the FVCOM object and 'flow' option to 5.0 from the sea surface downwards.
# - Initialize ***val3***, a Validation object using as input a list of all the measurement objects and the FVCOM object and 'flow' option to 'daf' using the ***harmonic_reconstruct*** option on.
# 
# ***Answer: ***

# In[4]:


drift = Drifter('./data4tutorial/drifter_GP_01aug2013.mat')
adcp = ADCP('./data4tutorial/adcp_GP_01aug2013.mat')
tg = TideGauge('./data4tutorial/tidegauge_GP_01aug2013.mat')
station = Station('http://ecoii.acadiau.ca/thredds/dodsC/ecoii/test/Station3D_dngrid_BF_20130730_20130809.nc')
fvcom = FVCOM('http://ecoii.acadiau.ca/thredds/dodsC/ecoii/test/FVCOM3D_dngrid_GP_20130801_20130802.nc')
val1 = Validation(adcp, station, flow=10.0)
val2 = Validation([adcp, drift, tg], fvcom, flow='daf')
val3 = Validation(adcp, fvcom, flow=-5.0)


# ### 1.3. Object attributes, functions, methods & special methods

# The Validation object possesses 2 attributes and 5 methods. They would appear by typing ***val1. Tab*** for instance.
# 
# An *attribute* is a quantity intrinsic to its *object*. A *method* is an intrinsic *function* which changes an *attribute* of its object. Contrarily a *function* will generate its own *output*:
object.method(inputs)
output = object.function(inputs)
# The Validation attributes are:
# - ***History***: history metadata that keeps track of the object changes
# - ***Variables***: gathers the hydrodynamics related data. Note that methods will generate new fields in this attribute
# - ***Benchmarks***: gathers the standard validation benchmarks. Note that this attribute will only appear once one of the **validate** methods has been ran. 
# 
# The Validation methods & functions are:
# - ***benchmarks_map***: plots bathymetric map & model validation benchmarks
# - ***validate_data***: computes series of standard validation benchmarks based on time-series analysis
# - ***validate_harmonics***: This method computes and store in a csv file the error in % for each component of the harmonic analysis (i.e. *_error.csv). 
# - ***taylor_diagram***: plots Taylor diagram based on the results of 'validate_data'
# - ***Write validation_report***: This method writes a report (*.pdf) based on the validation methods' results
# - ***save_as***: saves the current Validation structure into a *pickle* or a *mat* file

# ## 2. PySeidon-dvt - Hands-on (15 minutes)

# ***Exercise 3: ***
# - Perform ***data validation*** of **val1** with the **plot** option to True, **save_csv** option to True and **filename** to 'test1'.
# - ***Notice*** that folder **adcp_GP_01aug2013** has been created.
# - Plot the val1's ***taylor diagram***
# - Perform ***harmonic validation*** of **val1** with *filename* option = 'test2'.
# - ***Note*** that **test_el_harmo_error.csv** and **test_vel0_harmo_error.csv** have been created. 
# 
# ***Answer: ***

# In[5]:


val1.validate_data(plot=True, save_csv=True, filename='test1')
val1.taylor_diagram()
val1.validate_harmonics(filename='test2')


# ***Exercise 4: ***
# - Perform ***data validation*** of **val2** with **save_csv** option to True and **filename** to 'test3'. **Note**: it takes some time...
# - ***Notice*** that folders **adcp_GP_01aug2013_bis**, **drifter_GP_01aug2013** and **tidegauge_GP_01aug2013** have been created.
# - Plot the val2's ***taylor diagram***
# 
# ***Answer: ***

# In[6]:


val2.validate_data(save_csv=True, filename='test3')
val2.taylor_diagram()


# ***Exercise 5: ***
# - Perform ***data validation*** of **val3** with **save_csv** option to True , **phase_shift** correction to True and **filename** to 'test4'. **Note**: it takes some time...
# - ***Notice*** that folders **adcp_GP_01aug2013_bis_bis** has been created.
# - Plot the val3's ***benchmark map***
# - ***Write validation report*** of val3

# In[7]:


val3.validate_data(save_csv=True, phase_shift=True, filename='test4')
val3.benchmarks_map()
val3.write_validation_report(report_title="val3_report.pdf")


# ## 3. Validation benchmark definitions

# This collection and analysis of a set of statistics mostly adhere to the benchmarks defined as standards for
# hydrodynamic model validation by NOAA [1]. Additional statistics have been added to provide additional clarity on the skill of the model [2, 3, 4].
# 
# The present validation set is usually performed the following variables yet could be extended to other hydrodynamic quantities:
# - Current speed (unsigned velocity)
# - Current direction (between -180 and 180 degrees)
# - Elevation (deviation from mean sea level)
# 
# ***Benchmarks***
# 
# Following is a list of the statistics used to evaluate model skil
# - RMSE: Root Mean Squared Error
# - SD: Standard Deviation of Error
# - CF(X): Central Frequency; percentage of error values that fall within the range (-X, X)
# - POF(X): Positive Outlier Frequency; percentage of error values that fall above X
# - NOF(X): Negative Outlier Frequency; percentage of error values that fall below -X
# - MDPO(X): Maximum Duration of Positive Outliers; longest number of minutes during which consecutive errors fall above X
# - MDNO(X): Maximum Duration of Negative Outliers; longest number of minutes during which consecutive errors fall below X
# - Willmott Skill: A measure of model adherence to observed data between 0 and 1, with 0 being absolutely no adherence, and 1 being perfect.
# - Phase: The phase shift (minutes) of model data that minimizes RMSE across a timespan of +/-3hr.
# - R2 (i.e. coefficient of determination): Measure of the strength of the linear correlation between the model data and the observed data between 0 and 1, with 0 being no correlation, and 1 being perfect correlation.
# 
# The value of X is dependent on the variable and the statistic it's being used for. As defined here, X equals 10% of the data range. NOAA additionally defines limits on the values for determining the skill of the model. Accordingly, CF can be no lower than 90%, NOF/POF can be no higher than 1%, and MDPO/MDNO can be no longer than 24hr/1440min. ideally, the
# Willmott Skill and R2 should be 1, and the phase should be 0.
# 
# ***References***
# 
# [1] K. W. Hess, T. F. Gross, R. A. Schmalz, J. G. Kelley, F. Aikman and E. Wei, "NOS standards for evaluating
# operational nowcast and forecst hydrodynamic model systems," National Oceanic and Atmospheric
# Administration, Silver Srping, Maryland, 2003.
# 
# [2] K. Gunn and C. Stock-Williams, "On validating numerical hydrodynamic models of complex tidal flow,"
# International Journal of Marine Energy, Vols. 3-4, no. Special, pp. 82-97, 2013.
# 
# [3] N. Georgas and A. F. Blumberg, "Establishing Confidence in Marine Forecast Systems: The Design and Skill
# Assessment of the New York Harbor Observation and Prediction System, Version 3 (NYHOPS v3)," in 11th
# International Conference on Estuarine and Coastal Modeling, Seattle, Washington, United States, 2010.
# 
# [4] Y. Liu, P. MacCready, H. M. Barbara, E. P. Dever, M. Kosro and N. S. Banas, "Evaluation of a coastal ocean
# circulation model for the Columbia River plume in summer 2004," Journal of Geophysical Research, vol. 114,
# no. C2, p. 1978–2012, 2009.

# ## 4. Bug patrol & steering committee

# ### 4.1. Bug report

# As beta tester, your first assignement is to report bugs...yet not everything is a bug. The first thing to check before to report a bug is to verify that your version of *PySeidon-dvt* is up-to-date. The best way to keep up with the package evolution is to [***git***](http://git-scm.com/) to ***clone*** the repository, use ***pull*** to update it and ***re-install*** it if needed. 
# 
# The second thing to check before to report a bug is to verify that the bug is ***reproducible***. When running into a bug, double check that your inputs fit the description of the documentation then turn the ***debug flag on*** (e.g. *output = tidegaugeobject.function(inputs, debug=True)*) and submit the command again. If the error re-occurs then report it (i.e. copy entire error message + command and send it to package administrator)

# ### 4.2. Suggestions & critics

# Your second role as beta-tester is to submit suggestions and critics to the developpers regarding the functioning and functionality of the package. Beta testing phase is the best opportunity to steer a project towards the applications you would like to be tackled...