Pass multiple parameters to concurrent.futures.Executor.map? Pass multiple parameters to concurrent.futures.Executor.map? python python

Pass multiple parameters to concurrent.futures.Executor.map?


One argument that is repeated, one argument in c

from itertools import repeatfor result in executor.map(f, repeat(a), c):    pass

Need to unpack items of c, and can unpack c

from itertools import izipfor result in executor.map(f, *izip(*c)):    pass

Need to unpack items of c, can't unpack c

  1. Change f to take a single argument and unpack the argument in the function.
  2. If each item in c has a variable number of members, or you're calling f only a few times:

    executor.map(lambda args, f=f: f(*args), c)

    It defines a new function that unpacks each item from c and calls f. Using a default argument for f in the lambda makes f local inside the lambda and so reduces lookup time.

  3. If you've got a fixed number of arguments, and you need to call f a lot of times:

    from collections import dequedef itemtee(iterable, n=2):    def gen(it = iter(iterable), items = deque(), next = next):        popleft = items.popleft        extend = items.extend        while True:            if not items:                extend(next(it))            yield popleft()    return [gen()] * nexecutor.map(f, *itemtee(c, n))

Where n is the number of arguments to f. This is adapted from itertools.tee.


You need to remove the * on the map call:

args = ((a, b) for b in c)for result in executor.map(f, args):    pass

This will call f, len(args) times, where f should accept one parameter.

If you want f to accept two parameters you can use a lambda call like:

args = ((a, b) for b in c)for result in executor.map(lambda p: f(*p), args):   # (*p) does the unpacking part    pass


You can use currying to create new function via partial method in Python

from concurrent.futures import ThreadPoolExecutorfrom functools import partialdef some_func(param1, param2):    # some code# currying some_func with 'a' argument is repeatedfunc = partial(some_func, a)with ThreadPoolExecutor() as executor:    executor.map(func, list_of_args):    ...

If you need to pass more than one the same parameters you can pass them to partial method

func = partial(some_func, a, b, c)