Set up a logger, so we can actually see the output (this is unnecessary, but helpful for debugging):
import logging
logger = logging.Logger('ipcluster')
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler(sys.stdout))
Get some config that the launchers expect from ipcluster:
ip = get_ipython()
config = ip.config
profile_dir = ip.profile_dir.location # aka ipython_dir/profile_{name}
Import the Launcher classes used for starting controller/engine subprocesses
from IPython.parallel.apps.launcher import LocalControllerLauncher, LocalEngineLauncher
There is also a LocalEngineSetLauncher class that allows you to start/stop a group of engines at a time
Define some functions for simple start/stop of engines
import time
engines = []
def add_engines(n, delay=0.1):
"""start n engines"""
for i in range(n):
e = LocalEngineLauncher(config=config, log=logger, profile_dir=profile_dir)
engines.append(e)
e.start()
if i + 1 < n:
time.sleep(delay)
def cleanup_engines():
[ e.stop() for e in engines ]
Start the Controller:
controller = LocalControllerLauncher(config=config, log=logger, profile_dir=profile_dir)
controller.start()
Start 5 engines:
add_engines(5)
Now we can work with the engines in the usual way:
from IPython import parallel
rc = parallel.Client()
rc.ids
rc[:]['a'] = 5
rc[:]['a']
And we can stop them as well:
engines[0].stop()
rc.ids
And add some more
add_engines(3)
rc.ids
And cleanup the whole cluster:
rc.shutdown(hub=True)