#!/usr/bin/env python
# coding: utf-8

# > This is one of the 100 recipes of the [IPython Cookbook](http://ipython-books.github.io/), the definitive guide to high-performance scientific computing and data science in Python.

# # 5.10. Interacting with asynchronous parallel tasks in IPython

# You need to start IPython engines (see previous recipe). The simplest option is to launch them from the *Clusters* tab in the notebook dashboard. In this recipe, we use four engines.

# 1. Let's import a few modules.

# In[ ]:


import time
import sys
from IPython import parallel
from IPython.display import clear_output, display
from IPython.html import widgets


# 2. We create a Client.

# In[ ]:


rc = parallel.Client()


# 3. Now, we create a load balanced view on the IPython engines.

# In[ ]:


view = rc.load_balanced_view()


# 4. We define a simple function for our parallel tasks.

# In[ ]:


def f(x):
    import time
    time.sleep(.1)
    return x*x


# 5. We will run this function on 100 integer numbers in parallel.

# In[ ]:


numbers = list(range(100))


# 6. We execute `f` on our list `numbers` in parallel across all of our engines, using `map_async()`. This function returns immediately an `AsyncResult` object. This object allows us to retrieve interactively information about the tasks.

# In[ ]:


ar = view.map_async(f, numbers)


# 7. This object has a `metadata` attribute, a list of dictionaries for all engines. We can get the date of submission and completion, the status, the standard output and error, and other information.

# In[ ]:


ar.metadata[0]


# 8. Iterating over the `AsyncResult` instance works normally; the iteration progresses in real-time while the tasks are being completed.

# In[ ]:


for _ in ar:
    print(_, end=', ')


# 9. Now, we create a simple progress bar for our asynchronous tasks. The idea is to create a loop polling for the tasks' status at every second. An `IntProgressWidget` widget is updated in real-time and shows the progress of the tasks.

# In[ ]:


def progress_bar(ar):
    # We create a progress bar.
    w = widgets.IntProgressWidget()
    # The maximum value is the number of tasks.
    w.max = len(ar.msg_ids)
    # We display the widget in the output area.
    display(w)
    # Repeat every second:
    while not ar.ready():
        # Update the widget's value with the
        # number of tasks that have finished
        # so far.
        w.value = ar.progress
        time.sleep(1)
    w.value = w.max


# In[ ]:


ar = view.map_async(f, numbers)


# In[ ]:


progress_bar(ar)


# 10. Finally, it is easy to debug a parallel task on an engine. We can launch a Qt client on the remote kernel by calling `%qtconsole` within a `%%px` cell magic.

# In[ ]:


get_ipython().run_cell_magic('px', '-t 0', '%qtconsole\n')


# The Qt console allows us to inspect the remote namespace for debugging or analysis purposes.

# > You'll find all the explanations, figures, references, and much more in the book (to be released later this summer).
# 
# > [IPython Cookbook](http://ipython-books.github.io/), by [Cyrille Rossant](http://cyrille.rossant.net), Packt Publishing, 2014 (500 pages).