%pylab inline
Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.kernel.zmq.pylab.backend_inline]. For more information, type 'help(pylab)'.
import time
import numpy as np
from IPython.parallel import Client
rc = Client()
def f(x):
import random, time
time.sleep(3 * random.random())
return x
lview = rc.load_balanced_view()
Time just the task submission
def scale_submit(view, f, nlist):
sends = []
for n in nlist:
time.sleep(1)
x = range(n)
tic = time.time()
amr = view.map_async(f, x)
sent = time.time() - tic
print " sent %6i tasks in %6i ms" % (n, 1000 * sent)
sys.stdout.flush()
sends.append(sent)
return sends
nlist = [ 2 ** p for p in range(1,16) ]
nlist
[2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768]
def plot_tasks_per_sec(ns, ts):
ns = np.array(ns)
ts = np.array(ts)
plt.loglog(ns, ns/ts, '-o', label="measured")
# plt.ylim(0,5e3)
plt.xlabel("n")
plt.ylabel("tasks/sec")
This is run on master:
nlist = nlist[:-2]
sent = scale_submit(lview, f, nlist)
plot_tasks_per_sec(nlist, sent)
sent 2 tasks in 2 ms sent 4 tasks in 4 ms sent 8 tasks in 16 ms sent 16 tasks in 12 ms sent 32 tasks in 25 ms sent 64 tasks in 45 ms sent 128 tasks in 100 ms sent 256 tasks in 268 ms sent 512 tasks in 523 ms sent 1024 tasks in 1428 ms sent 2048 tasks in 4326 ms sent 4096 tasks in 16289 ms sent 8192 tasks in 70582 ms
Oh noes, that's crazy slow!
And this is run on PR #3108
sent = scale_submit(lview, f, nlist)
plot_tasks_per_sec(nlist, sent)
sent 2 tasks in 2 ms sent 4 tasks in 3 ms sent 8 tasks in 6 ms sent 16 tasks in 13 ms sent 32 tasks in 25 ms sent 64 tasks in 45 ms sent 128 tasks in 86 ms sent 256 tasks in 182 ms sent 512 tasks in 341 ms sent 1024 tasks in 708 ms sent 2048 tasks in 1416 ms sent 4096 tasks in 2782 ms sent 8192 tasks in 5505 ms sent 16384 tasks in 11259 ms sent 32768 tasks in 23846 ms
Hooray, flat scaling!