Python, using multiprocess is slower than not using it

This example is too small to benefit from multiprocessing.

There's a LOT of overhead when starting a new process. If there were heavy processing involved, it would be negligable. But your example really isn't all that intensive, and so you're bound to notice the overhead.

You'd probably notice a bigger difference with real threads, too bad python (well, CPython) has issues with CPU-bound threading.

Multiprocessing could be useful for what you're doing, but not in the way you're thinking about using it. As you're basically doing some computation on every member of a list, you could do it using the multiprocessing.Pool.map method, to do the computation on the list members in parallel.

Here is an example that shows your code's performance using a single process and using multiprocessing.Pool.map:

from multiprocessing import Poolfrom random import choicefrom string import printablefrom time import timedef build_test_list():    # Builds a test list consisting of 5 sublists of 10000 strings each.    # each string is 20 characters long    testlist = [[], [], [], [], []]    for sublist in testlist:        for _ in xrange(10000):            sublist.append(''.join(choice(printable) for _ in xrange(20)))    return testlistdef process_list(l):    # the time-consuming code    result = []    tmp = []    for n in range(len(l)):        if l[n] not in tmp:            result.insert(n, l[n]+' ('+str(l.count(l[n]))+')')            tmp.insert(0, l[n])    return resultdef single(l):    # process the test list elements using a single process    results = []    for sublist in l:        results.append(process_list(sublist))    return resultsdef multi(l):    # process the test list elements in parallel    pool = Pool()    results = pool.map(process_list, l)    return resultsprint "Building the test list..."testlist = build_test_list()print "Processing the test list using a single process..."starttime = time()singleresults = single(testlist)singletime = time() - starttimeprint "Processing the test list using multiple processes..."starttime = time()multiresults = multi(testlist)multitime = time() - starttime# make sure they both return the same thingassert singleresults == multiresultsprint "Single process: {0:.2f}sec".format(singletime)print "Multiple processes: {0:.2f}sec".format(multitime)

Output:

Building the test list...Processing the test list using a single process...Processing the test list using multiple processes...Single process: 34.73secMultiple processes: 24.97sec

ETA: Now that you've posted your code, I can tell you there is a simple way to do what you're doing MUCH faster (>100 times faster).

I see that what you're doing is adding a frequency in parentheses to each item in a list of strings. Instead of counting all the elements each time (which, as you can confirm using cProfile, is by far the largest bottleneck in your code), you can just create a dictionary that maps from each element to its frequency. That way, you only have to go through the list twice- once to create the frequency dictionary, once to use it to add frequency.

Here I'll show my new method, time it, and compare it to the old method using a generated test case. The test case even shows the new result to be exactly identical to the old one. Note: All you really need to pay attention to below is the new_method.

import randomimport timeimport collectionsimport cProfileLIST_LEN = 14000def timefunc(f):    t = time.time()    f()    return time.time() - tdef random_string(length=3):    """Return a random string of given length"""    return "".join([chr(random.randint(65, 90)) for i in range(length)])class Profiler:    def __init__(self):        self.original = [[random_string() for i in range(LIST_LEN)]                            for j in range(4)]    def old_method(self):        self.ListVar = self.original[:]        for b in range(len(self.ListVar)):            self.list1 = []            self.temp = []            for n in range(len(self.ListVar[b])):                if not self.ListVar[b][n] in self.temp:                    self.list1.insert(n, self.ListVar[b][n] + '(' +    str(self.ListVar[b].count(self.ListVar[b][n])) + ')')                    self.temp.insert(0, self.ListVar[b][n])            self.ListVar[b] = list(self.list1)        return self.ListVar    def new_method(self):        self.ListVar = self.original[:]        for i, inner_lst in enumerate(self.ListVar):            freq_dict = collections.defaultdict(int)            # create frequency dictionary            for e in inner_lst:                freq_dict[e] += 1            temp = set()            ret = []            for e in inner_lst:                if e not in temp:                    ret.append(e + '(' + str(freq_dict[e]) + ')')                    temp.add(e)            self.ListVar[i] = ret        return self.ListVar    def time_and_confirm(self):        """        Time the old and new methods, and confirm they return the same value        """        time_a = time.time()        l1 = self.old_method()        time_b = time.time()        l2 = self.new_method()        time_c = time.time()        # confirm that the two are the same        assert l1 == l2, "The old and new methods don't return the same value"        return time_b - time_a, time_c - time_bp = Profiler()print p.time_and_confirm()

When I run this, it gets times of (15.963812112808228, 0.05961179733276367), meaning it's about 250 times faster, though this advantage depends on both how long the lists are and the frequency distribution within each list. I'm sure you'll agree that with this speed advantage, you probably won't need to use multiprocessing :)

(My original answer is left in below for posterity)

ETA: By the way, it is worth noting that this algorithm is roughly linear in the length of the lists, while the code you used is quadratic. This means it performs with even more of an advantage the larger the number of elements. For example, if you increase the length of each list to 1000000, it takes only 5 seconds to run. Based on extrapolation, the old code would take over a day :)

It depends on the operation you are performing. For example:

import timeNUM_RANGE = 100000000from multiprocessing  import Processdef timefunc(f):    t = time.time()    f()    return time.time() - tdef multi():    class MultiProcess(Process):        def __init__(self):            Process.__init__(self)        def run(self):            # Alter string + test processing speed            for i in xrange(NUM_RANGE):                a = 20 * 20    thread1 = MultiProcess()    thread2 = MultiProcess()    thread1.start()    thread2.start()    thread1.join()    thread2.join()def single():    for i in xrange(NUM_RANGE):        a = 20 * 20    for i in xrange(NUM_RANGE):        a = 20 * 20print timefunc(multi) / timefunc(single)

On my machine, the multiprocessed operation takes up only ~60% the time of the singlethreaded one.