Create our Client
from IPython import parallel
client = parallel.Client()
lbv = client.load_balanced_view()
dview = client[:]
dview
<DirectView [0, 1]>
Interactively defined classes can be annoying to use with IPython.parallel. This stems from how class instances are serialized by pickle
# the real model object is much more involved...
class MyModel(object):
def maximize(self, r):
return max(r, 4)
def evaluate(r, model):
return model.maximize(r)
import cPickle as pickle
m = MyModel()
print pickle.dumps(m)
ccopy_reg _reconstructor p1 (c__main__ MyModel p2 c__builtin__ object p3 NtRp4 .
Note how the pickled model is almost nothing but a reference to __main__.MyModel
and object
.
This is how the instance will be reconstructed remotely. But the probem is that MyModel is not defined on the engines:
model = MyModel()
lbv.apply_sync(evaluate, 2, model)
--------------------------------------------------------------------------- AttributeError Traceback (most recent call last)/Users/minrk/dev/ip/mine/IPython/kernel/zmq/serialize.pyc in unpack_apply_message(bufs, g, copy) 191 args = [] 192 for i in range(info['nargs']): --> 193 arg, arg_bufs = unserialize_object(arg_bufs, g) 194 args.append(arg) 195 args = tuple(args) /Users/minrk/dev/ip/mine/IPython/kernel/zmq/serialize.pyc in unserialize_object(buffers, g) 130 # a zmq message 131 pobj = bytes(pobj) --> 132 canned = pickle.loads(pobj) 133 if istype(canned, sequence_types) and len(canned) < MAX_ITEMS: 134 for c in canned: AttributeError: 'DummyMod' object has no attribute 'MyModel'
This is a common problem with relying on interactively defined names, but there are issues peculiar to classes, as opposed to locally defined functions.
In IPython 0.13, there are a few shortcomings:
%%px
to define a class both locally and remotely.So defining a class everywhere is a two-step process, either manual:
class MyModel1(object):
def maximize(self, r):
return max(r, 0)
%%px
class MyModel1(object):
def maximize(self, r):
return max(r, 0)
model = MyModel1()
lbv.apply_sync(evaluate, 1, model)
1
Or more automatic:
class MyModel2(object):
def maximize(self, r):
return max(r, 2)
# this just executes the previous cell on all our engines
# (note that In[-1] is *this* cell, so In[-2] is the previous one)
dview.execute(In[-2], block=True)
<AsyncResult: finished>
model = MyModel2()
lbv.apply_sync(evaluate, 1, model)
2
In IPython 1.0, this has been alleviated to some degree, by making interactively defined classes pushable,
and adding %%px --local
.
So now, you can use the same push step that is commonly used when you have multi-function tasks:
class MyModel3(object):
def maximize(self, r):
return max(r, 3)
dview['MyModel3'] = MyModel3
model = MyModel3()
lbv.apply_sync(evaluate, 0, model)
3
Or you can simultaneously define the class both locally and remotely with %%px --local
%%px --local
# the real model object is much more involved...
class MyModel4(object):
def maximize(self, r):
return max(r, 4)
model = MyModel4()
lbv.apply_sync(evaluate, 2, model)
4
Now, if you really want bleeding edge, there is an outstanding Pull Request that lets you define dependencies for locally defined names, which will actually attach the local objects to the task, so they are guaranteed to be defined when the task runs
class MyModel5(object):
def maximize(self, r):
return max(r, 5)
@parallel.require(MyModel5)
def evaluate(r, model):
return model.maximize(r)
model = MyModel5()
lbv.apply_sync(evaluate, 2, model)
5