CPU Affinity lets you pin processes to particular processors on your machine. The Python package psutil lets you view and edit the CPU affinity of any process.
import os
import psutil
p = psutil.Process(os.getpid())
p.get_cpu_affinity()
[0, 1, 2, 3]
Setting CPU affinity can be a problem if you aren't aware that it is happening. The most common way for CPU affinity to be set without the user being aware is importing numpy linked against particular BLAS libraries.
The most common symptom of this will be that all of your IPython engines will use just one core.
Let's check if this happens here:
import numpy
p.get_cpu_affinity()
[0, 1, 2, 3]
Okay, so we are fine. There are a few logical cases where we want to set the CPU affinity of our engines.
from IPython import parallel
rc = parallel.Client()
dview = rc[:]
rc.ids
[0, 1, 2, 3]
This assumes that you already only have one engine per core. In this case, I have four.
dview.scatter("cpu", range(len(dview)), flatten=True)
%px print cpu
[stdout:0] 0 [stdout:1] 1 [stdout:2] 2 [stdout:3] 3
Now we can set the affinity of each engine to a single CPU core, so the engines won't be fighting over CPU resources.
%%px
import os
import psutil
p = psutil.Process(os.getpid())
p.set_cpu_affinity([cpu])
print p.get_cpu_affinity()
[stdout:0] [0] [stdout:1] [1] [stdout:2] [2] [stdout:3] [3]
A simpler case is just restoring CPU affinity after some library may have set it inappropriately.
We use the multiprocessing.cpu_count
to get the number of CPUs.
%%px
import os
import psutil
from multiprocessing import cpu_count
p = psutil.Process(os.getpid())
p.set_cpu_affinity(range(cpu_count()))
print p.get_cpu_affinity()
[stdout:0] [0, 1, 2, 3] [stdout:1] [0, 1, 2, 3] [stdout:2] [0, 1, 2, 3] [stdout:3] [0, 1, 2, 3]