Using line profiler with multiprocessing

Question

How can you profile a python module that use multiprocessing (multiprocessing.Pool.map) so each spawned process will be also profiled line by line.

Currently I use line_profiler for profiling but it doesn't support multiprocessing. Is there a way to do it manually? Or maybe use some other tool?

Answer 1

The normal way of using line_profiler of adding @profile to the function being profiled and running kernprof -v -l script.py leads to the following error for multiprocessing:

Can't pickle <class '__main__.Worker'>: attribute lookup Worker on __main__ failed.

To fix this, we have to setup the line_profiler ourselves in the sub-process we want to profile, rather than doing it globally via kernelprof:

import multiprocessing as mp
import line_profiler

class Worker(mp.Process):

    def run(self):
        prof = line_profiler.LineProfiler()
        # Wrap all functions that you want to be profiled in this process
        # These can be global functions or any class methods
        # Make sure to replace instance methods on a class level, not the bound methods self.run2
        Worker.run2 = prof(Worker.run2)
        ...
        # run the main
        self.run2()
        # store stats in separate file for each process
        prof.dump_stats('worker.lprof')

    def run2(self):
        # real run method renamed
        ...

Now running the script this generates a profile file that we can then visualize with:

python -m line_profiler worker.lprof