IPython has a few magics for working with your engines.
This assumes you have started an IPython cluster, either with the notebook interface,
from IPython import parallel rc = parallel.Client() dv = rc[:] rc.ids
Creating a Client registers the parallel magics
These magics are initially associated with a DirectView always associated with all currently registered engines.
Now we can execute code remotely with
%px print a
with dv.sync_imports(): import sys
%px print >> sys.stderr, "ERROR"
You don't have to wait for results. The
%pxconfig magic lets you change the default blocking/targets for the
%px import time %px time.sleep(5) %px time.time()
But you will notice that this didn't output the result of the last command.
For this, we have
%pxresult, which displays the output of the latest request:
Remember, an IPython engine is IPython, so you can do magics remotely as well!
%pxconfig --block %px %pylab inline
%%px can also be used as a cell magic, for submitting whole blocks.
This one acceps
--noblock flags to specify
the blocking behavior, though the default is unchanged.
dv.scatter('id', dv.targets, flatten=True) dv['stride'] = len(dv)
%%px --noblock x = linspace(0,pi,1000) for n in range(id,12, stride): print n plt.plot(x,sin(n*x)) plt.title("Plot %i" % id)
It also lets you choose some amount of the grouping of the outputs with
The choices are:
engine- all of an engine's output is collected together
type- where stdout of each engine is grouped, etc. (the default)
order- same as
type, but individual displaypub outputs are interleaved. That is, it will output the first plot from each engine, then the second from each, etc.
%%px --group-outputs=engine x = linspace(0,pi,1000) for n in range(id+1,12, stride): print n plt.figure() plt.plot(x,sin(n*x)) plt.title("Plot %i" % n)
When you specify 'order', then individual display outputs (e.g. plots) will be interleaved.
%pxresult takes the same output-ordering arguments as
so you can view the previous result in a variety of different ways with a few sequential calls to
When a DirectView has a single target, the output is a bit simpler (no prefixes on stdout/err, etc.):
def generate_output(): """function for testing output publishes two outputs of each type, and returns something """ import sys,os from IPython.core.display import display, HTML, Math print "stdout" print >> sys.stderr, "stderr" display(HTML("<b>HTML</b>")) print "stdout2" print >> sys.stderr, "stderr2" display(Math(r"\alpha=\beta")) return os.getpid() dv['generate_output'] = generate_output
You can also have more than one set of parallel magics registered at a time.
View.activate() method takes a suffix argument, which is added to
e0 = rc[-1] e0.block = True e0.activate('0')
As mentioned above, we can redisplay those same results with various grouping:
%pxresult --group-outputs order
%pxresult --group-outputs engine
When you raise exceptions with the parallel exception, the CompositeError raised locally will display your remote traceback.
%%px from numpy.random import random A = random((100,100,'invalid shape'))
Remember, Engines are IPython too, so the cell that is run remotely by %%px can in turn use a cell magic.
%%px %%timeit from numpy.random import random from numpy.linalg import norm A = random((100,100)) norm(A, 2)