#!/usr/bin/env python
# coding: utf-8

# We presented at IJCNN, 2015 the following paper, which won the [Best Paper Award](http://www.ijcnn.org/assets/docs/ijcnn2015-awards.pdf)
# 
#   * Anderson, C., Lee, M., and Elliott, D., "[Faster Reinforcement Learning After Pretraining Deep Networks to Predict State Dynamics](publications/pretrainijcnn15.pdf)", Proceedings of the IJCNN, 2015, Killarney, Ireland.
# 
# *Abstract:* Deep learning algorithms have recently appeared that pretrain hidden layers of neural networks in unsupervised ways, leading to state-of-the-art performance on large classification problems. These methods can also pretrain networks used for reinforcement learning. However, this ignores the additional information that exists in a reinforcement learning paradigm via the ongoing sequence of state, action, new state tuples. This paper demonstrates that learning a predictive model of state dynamics can result in a pretrained hidden layer structure that reduces the time needed to solve reinforcement learning problems. 

# Below is a sequence of animations showing the progress of reinforcement learning.  We simulated the dynamics of the cart-pole problem with full 360 pole rotation and collisions of the cart at the ends of the track. Details of this simulation will be provided here soon.

# <video width="400" controls src="http://www.cs.colostate.edu/~anderson/wp/wp-content/uploads/2017/06/pole-2015-0min.mp4">

# The animation above shows that after a few steps, our RL agent seems to prefer pushes to the right.

# <video controls width="400" src="../wp-content/uploads/2017/06/pole-2015-10min.mp4">

# After 10 minutes of training, it is starting to move away from the right edge.

# <video controls width="400" src="videos/pole-2015-50min.mp4">

# After 50 minutes of training, the pole is swinging up, but not being balanced.

# <video controls width="400" src="videos/pole-2015-100min.mp4">

# After 100 minutes of training, our RL agent occasionally is able to use bumps at end of track to help swing the pole up.

# <video controls width="400" src="videos/pole-2015-200min.mp4">

# And finally, after 200 minutes, the pole is being balanced!

# <video controls width="400" src="videos/pole-2015-test1.mp4">

# With $\epsilon = 0$, so no exploration, we see quick swing up and balance.

# <video controls width="400" src="videos/pole-2015-test1-slow.mp4">

# Again, in slow motion.

# <video controls width="400" src="videos/pole-2015-test2-slow.mp4">

# Here is another run with $\epsilon = 0$, in slow motion.