How to run three functions at the same time (and return values from each)?

Question

I have three functions, each returning a list. The problem is that running each function takes around 20-30 seconds. So running the entire script ends up taking about 2 min.

I want to use multiprocessing or multithreading (whichever is easier to implement) to have all three functions running at the same time.

The other hurdle I ran into was I that I'm not sure how to return the list from each of the functions.

def main():
    masterlist = get_crs_in_snow()
    noop_crs = get_noops_in_snow()
    made_crs = get_crs_in_git()

    # take the prod master list in SNOW, subtract what's been made or is in the noop list
    create_me = [obj for obj in masterlist if obj not in made_crs and obj not in noop_crs]

    print "There are {0} crs in Service Now not in Ansible".format(len(create_me))
    for cr in create_me:
        print str(cr[0]),


if __name__ == '__main__':
    main()

I figure I can get some significant improvements in run time just by multithreading or multiprocessing the following line:

masterlist = get_crs_in_snow()
noop_crs = get_noops_in_snow()
made_crs = get_crs_in_git()

How do I have these three functions run at the same time?

Answer 1

This is completely untested since I don't have the rest of your code, but it may give you an idea of what can be done. I have adapted your code into the multiprocessing pattern:

from multiprocessing import Pool

def dispatcher(n):
    if n == 0:
        return get_crs_in_snow()
    if n == 1:
        return get_noops_in_snow()
    if n == 2:
        return get_crs_in_git()

def main():
    pool = Pool(processes=3)

    v = pool.map(dispatcher, range(3))

    masterlist = v[0]
    noop_crs = v[1]
    made_crs = v[2]

    # take the prod master list in SNOW, subtract what's been made or is in the noop list
    create_me = [obj for obj in masterlist if obj not in made_crs and obj not in noop_crs]

    print "There are {0} crs in Service Now not in Ansible".format(len(create_me))
    for cr in create_me:
        print str(cr[0]),


if __name__ == '__main__':
    main()