IPython.parallel doesn't do much in the way of serialization. It has custom zero-copy handling of numpy arrays, but other than that, it doesn't do anything other than the bare minimum to make basic interactively defined functions and classes sendable.
There are a few projects that extend pickle to make just about anything sendable, and one of these is dill.
NOTE: This requires IPython 1.0-dev and latest dill, currently only available from:
pip install dill-0.2a.dev-20120503.tar.gz
First, as always, we create our
def make_closure(a):
"""make a function with a closure, and return it"""
def has_closure(b):
return a * b
return has_closure
closer = make_closure(5)
closer(2)
10
import pickle
Without help, pickle can't deal with closures
pickle.dumps(closer)
--------------------------------------------------------------------------- PicklingError Traceback (most recent call last) <ipython-input-5-2dad7d7aaa8a> in <module>() ----> 1 pickle.dumps(closer) /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py in dumps(obj, protocol) 1372 def dumps(obj, protocol=None): 1373 file = StringIO() -> 1374 Pickler(file, protocol).dump(obj) 1375 return file.getvalue() 1376 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py in dump(self, obj) 222 if self.proto >= 2: 223 self.write(PROTO + chr(self.proto)) --> 224 self.save(obj) 225 self.write(STOP) 226 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py in save_global(self, obj, name, pack) 746 raise PicklingError( 747 "Can't pickle %r: it's not found as %s.%s" % --> 748 (obj, module, name)) 749 else: 750 if klass is not obj: PicklingError: Can't pickle <function has_closure at 0x10b6ac140>: it's not found as __main__.has_closure
But after we import dill, magic happens
import dill
pickle.dumps(closer)[:64] + '...'
"cdill.dill\n_load_type\np0\n(S'FunctionType'\np1\ntp2\nRp3\n(cdill.dill..."
So from now on, pretty much everything is pickleable.
As usual, we start by creating our Client and View
from IPython import parallel
rc = parallel.Client()
view = rc.load_balanced_view()
We can use dill to allow IPython.parallel to send anything.
Step 1. Disable IPython's special handling of function objects (nothing to work around anymore):
from types import FunctionType
from IPython.utils.pickleutil import can_map
can_map.pop(FunctionType, None)
IPython.utils.pickleutil.CannedFunction
Step 2. Switch the serialize module to use Python pickle instead of cPickle (dill extends pickle
to be able to dump anything, but this doesn't work with the optimized cPickle)
from IPython.kernel.zmq import serialize
serialize.pickle = pickle
And that's it! Now we can send closures and other previously non-pickleables to our engines.
view.apply_sync(closer, 3)
15
But wait, there's more!
view.apply_sync(make_closure, 2)
--------------------------------------------------------------------------- ValueError Traceback (most recent call last)/Users/minrk/dev/ip/mine/IPython/kernel/zmq/serialize.pyc in serialize_object(obj, buffer_threshold, item_threshold) 107 buffers.extend(_extract_buffers(cobj, buffer_threshold)) 108 --> 109 buffers.insert(0, pickle.dumps(cobj,-1)) 110 return buffers 111 /Users/minrk/dev/ip/mine/IPython/utils/codeutil.pyc in reduce_code(co) 32 def reduce_code(co): 33 if co.co_freevars or co.co_cellvars: ---> 34 raise ValueError("Sorry, cannot pickle code objects with closures") 35 args = [co.co_argcount, co.co_nlocals, co.co_stacksize, 36 co.co_flags, co.co_code, co.co_consts, co.co_names, ValueError: Sorry, cannot pickle code objects with closures
We can also apply very same patch to the engines, then we can send previously unpickleable objects both ways. Here's a way to do dill patch the pickle everywhere at once:
%%px --local
# load dill
import dill
# disable special function handling
from types import FunctionType
from IPython.utils.pickleutil import can_map
can_map.pop(FunctionType, None)
# fallback to pickle instead of cPickle, so that dill can take over
import pickle
from IPython.kernel.zmq import serialize
serialize.pickle = pickle
remote_closure = view.apply_sync(make_closure, 4)
remote_closure(5)
20
At this point, we can send/recv all kinds of stuff
def outer(a):
def inner(b):
def inner_again(c):
return c * b * a
return inner_again
return inner
So outer returns a function with a closure, which returns a function with a closure.
Now, we can resolve the first closure on the engine, the second here, and the third on a different engine, after passing through a lambda we define here and call there, just for good measure.
view.apply_sync(lambda f: f(3),view.apply_sync(outer, 1)(2))
6