executor.map and non-terating parameters

Question

I am trying to convert my script from using threads to much cooler multiprocessing (with python 3.2 and concurrent.futures, but this piece of code crashes

        with ThreadPoolExecutor(max_workers=MAX_THREADS) as executor:
            for result in executor.map(lambda h:
                                       validate_hostname(h, pci_ids, options.verbose),
                                       get_all_hostnames()):

I get error _pickle.PicklingError: Can't pickle <class 'function'>: attribute lookup builtins.function failed. When reading this answer I think the problem is that it is not possible to have lambda function as a parameter of executor.map() and in order to make executor.map() I would need to develop a one-parameter function, but those pci_ids and options.verbose are variable so I cannot specify them as fixed values in the help function.

Any ideas what to do?

Answer 1

To avoid the pickle error, you must define the function, validate, at the top-level of the module or script.

Since the function is being passed to executor.map it can only take one argument, so let that argument be a 3-tuple, (h, pci_ids, verbose).

def validate(arg):
    h, pci_ids, verbose = arg
    return validate_hostname(h, pci_ids, verbose)

with ThreadPoolExecutor(max_workers=MAX_THREADS) as executor:
    for result in executor.map(validate, [(host, pci_ids, options.verbose)
                                          for host in get_all_hostnames()]):