This is one of the 100 recipes of the IPython Cookbook, the definitive guide to high-performance scientific computing and data science in Python.
You need to start IPython engines (see previous recipe). The simplest option is to launch them from the Clusters tab in the notebook dashboard. In this recipe, we use four engines.
import time
import sys
from IPython import parallel
from IPython.display import clear_output, display
from IPython.html import widgets
rc = parallel.Client()
view = rc.load_balanced_view()
def f(x):
import time
time.sleep(.1)
return x*x
numbers = list(range(100))
f
on our list numbers
in parallel across all of our engines, using map_async()
. This function returns immediately an AsyncResult
object. This object allows us to retrieve interactively information about the tasks.ar = view.map_async(f, numbers)
metadata
attribute, a list of dictionaries for all engines. We can get the date of submission and completion, the status, the standard output and error, and other information.ar.metadata[0]
AsyncResult
instance works normally; the iteration progresses in real-time while the tasks are being completed.for _ in ar:
print(_, end=', ')
IntProgressWidget
widget is updated in real-time and shows the progress of the tasks.def progress_bar(ar):
# We create a progress bar.
w = widgets.IntProgressWidget()
# The maximum value is the number of tasks.
w.max = len(ar.msg_ids)
# We display the widget in the output area.
display(w)
# Repeat every second:
while not ar.ready():
# Update the widget's value with the
# number of tasks that have finished
# so far.
w.value = ar.progress
time.sleep(1)
w.value = w.max
ar = view.map_async(f, numbers)
progress_bar(ar)
%qtconsole
within a %%px
cell magic.%%px -t 0
%qtconsole
The Qt console allows us to inspect the remote namespace for debugging or analysis purposes.
You'll find all the explanations, figures, references, and much more in the book (to be released later this summer).
IPython Cookbook, by Cyrille Rossant, Packt Publishing, 2014 (500 pages).