How small do my tasks need to be (aka how fast is IPython)?
In parallel computing, an important relationship to keep in mind is the ratio of computation to communication. In order for your simulation to perform reasonably, you must keep this ratio high. When testing out a new tool like IPython, it is important to examine the limit of granularity that is appropriate. If it takes half a second of overhead to run each task, then breaking your work up into millisecond chunks isn't going to make sense.
Basic imports to use later, create a Client, and a LoadBalancedView of all the engines.
%matplotlib inline
import time
import numpy as np
from IPython.parallel import Client
rc = Client()
view = rc.load_balanced_view()
Sending and receiving tiny messages gives us a sense of the minimum time IPython must spend building and sending messages around. This should give us a sense of the minimum overhead of the communication system.
This should give us a sense of the lower limit on available granularity.
def test_latency(v, n):
tic = time.time()
echo = lambda x: x
tic = time.time()
for i in xrange(n):
v.apply_async(echo, '')
toc = time.time()
v.wait()
tac = time.time()
sent = toc-tic
roundtrip = tac-tic
return sent, roundtrip
for n in [8,16,32,64,128,256,512,1024]:
# short rest between tests
time.sleep(0.5)
s,rt = test_latency(view, n)
print "%4i %6.1f %6.1f" % (n,n/s,n/rt)
8 1336.0 233.0 16 1182.2 202.0 32 1378.5 271.1 64 1322.6 277.6 128 661.4 220.1 256 1218.5 278.8 512 1248.1 277.6 1024 1249.6 288.3
for n in [8,16,32,64,128,256,512,1024]:
# short rest between tests
time.sleep(0.5)
s,rt = test_latency(view, n)
print "%4i %6.1f %6.1f" % (n,n/s,n/rt)
8 969.3 97.8 16 1084.1 198.7 32 1251.3 261.6 64 795.2 208.2 128 831.3 282.1 256 667.4 172.6 512 438.0 165.9 1024 645.9 175.9
These tests were run on the loopback interface on a fast 8-core machine with 4 engines and slightly tuned non-default config (msgpack for serialization, TaskScheduler.hwm=0).
The hwm optimization is the most important for performance of these benchmarks.
The tests were done with the Python scheduler and pure-zmq scheduler, and with/without an SSH tunnel. We can see that the Python scheduler can do about 800 tasks/sec, while the pure-zmq scheduler gets an extra factor of two, at around 1.5k tasks/sec roundtrip. Purely outgoing - the time before the Client code can go on working, is closer to 4k msgs/sec sent. Using an SSH tunnel does not significantly impact performance, as long as you have a few tasks to line up.
Running the same test on a dedicated cluster with up to 128 CPUs shows that IPython does scale reasonably well.
Echoing numpy arrays is similar to the latency test, but scaling the array size instead of the number of messages tests the limits when there is data to be transferred.
def test_throughput(v, n, s):
A = np.random.random(s/8) # doubles are 8B
tic = time.time()
echo = lambda x: x
tic = time.time()
for i in xrange(n):
v.apply_async(echo, A)
toc = time.time()
v.wait()
tac = time.time()
sent = toc-tic
roundtrip = tac-tic
return sent, roundtrip
n = 128
for sz in [1e1,1e2,1e3,1e4,1e5,5e5,1e6,2e6]:
# short rest between tests
time.sleep(1)
s,rt = test_throughput(view, n, int(sz))
print "%8i %6.1f t/s %6.1f t/s %9.3f Mbps" % (sz,n/s,n/rt, 1e-6*sz*n/rt)
10 1125.6 t/s 285.6 t/s 0.003 Mbps 100 967.2 t/s 281.8 t/s 0.028 Mbps 1000 1246.3 t/s 281.6 t/s 0.282 Mbps 10000 1285.6 t/s 265.3 t/s 2.653 Mbps 100000 404.8 t/s 206.1 t/s 20.606 Mbps 500000 294.2 t/s 159.6 t/s 79.822 Mbps 1000000 328.4 t/s 108.8 t/s 108.802 Mbps 2000000 425.9 t/s 83.4 t/s 166.755 Mbps
n = 128
for sz in [1e1,1e2,1e3,1e4,1e5,5e5,1e6,2e6]:
# short rest between tests
time.sleep(1)
s,rt = test_throughput(view, n, int(sz))
print "%8i %6.1f t/s %6.1f t/s %9.3f Mbps" % (sz,n/s,n/rt, 1e-6*sz*n/rt)
10 1278.9 t/s 303.4 t/s 0.003 Mbps 100 1339.5 t/s 301.8 t/s 0.030 Mbps 1000 784.0 t/s 265.0 t/s 0.265 Mbps 10000 1083.7 t/s 274.0 t/s 2.740 Mbps 100000 336.1 t/s 143.6 t/s 14.357 Mbps 500000 289.4 t/s 150.9 t/s 75.453 Mbps 1000000 226.7 t/s 102.6 t/s 102.639 Mbps 2000000 368.9 t/s 70.2 t/s 140.495 Mbps
Note that the dotted lines, which measure the time it took to send the arrays is not a function of the message size. This is again thanks to pyzmq's non-copying sends. Locally, we can send 100 4MB arrays in ~50 ms, and libzmq will take care of actually transmitting the data while we can go on working.
Plotting the same data, scaled by message size shows that we are saturating the connection at ~1Gbps with ~10kB messages when using SSH, and ~10Gbps with ~50kB messages when not using SSH.
Another useful test is seeing how fast `view.map` is, for various numbers of tasks and for tasks of varying size.
These tests were done on AWS extra-large instances with the help of StarCluster, so the IO and CPU performance are quite low compared to a physical cluster.
def test_map(v,dt,n):
ts = [dt]*n
tic = time.time()
amr = v.map_async(time.sleep, ts)
toc = time.time()
amr.get()
tac = time.time()
sent = toc-tic
roundtrip = tac-tic
return sent, roundtrip
n = len(rc.ids) * 16
for dt in np.logspace(-3,0,7):
time.sleep(0.5)
s,rt = test_map(view, dt, n)
print "%4ims %5.1f%%" % (1000*dt, 1600*dt / rt)
1ms 7.6% 3ms 17.5% 10ms 63.1% 31ms 85.5% 100ms 95.8% 316ms 98.7% 1000ms 99.6%
This shows runs for jobs ranging from 1 to 128 ms, on 4,31,and 63 engines. On this system, millisecond jobs are clearly too small, but by the time individual tasks are > 100 ms, IPython overhead is negligible.
Now let's see how we use it for remote execution.