Nested list comprehension in Python

Since all the arrays (and sublists) in structure are the same size you can turn it into one higher dimensional array:

In [189]: A=np.array(structure)Out[189]: array([[[[ 1,  2,  3],         [ 4,  5,  6]],        [[ 7,  8,  9],         [10, 11, 12]]],       [[[13, 14, 15],         [16, 17, 18]],        [[19, 20, 21],         [22, 23, 24]]]])In [190]: A.shapeOut[190]: (2, 2, 2, 3)

Reshaping and swapaxes can give you all kinds of combinations.

For example, the values in your sample sublist can be selected with:

In [194]: A[:,0,:,:]Out[194]: array([[[ 1,  2,  3],        [ 4,  5,  6]],       [[13, 14, 15],        [16, 17, 18]]])

and reshape to get

In [197]: A[:,0,:,:].reshape(2,6)Out[197]: array([[ 1,  2,  3,  4,  5,  6],       [13, 14, 15, 16, 17, 18]])

and transpose to get the 6 rows of pairs:

In [198]: A[:,0,:,:].reshape(2,6).TOut[198]: array([[ 1, 13],       [ 2, 14],       [ 3, 15],       [ 4, 16],       [ 5, 17],       [ 6, 18]])

To get them in the 1,4,2,5.. order I can transpose first

In [208]: A[:,0,:,:].T.reshape(6,2)Out[208]: array([[ 1, 13],       [ 4, 16],       [ 2, 14],       [ 5, 17],       [ 3, 15],       [ 6, 18]])

Not sure exactly what full output you want but this may help:

imort numpy as npstructure = [[np.array([[1, 2, 3], [4, 5, 6]]), np.array([[7, 8, 9], [10, 11, 12]])],             [np.array([[13, 14, 15], [16, 17, 18]]), np.array([[19, 20, 21], [22, 23, 24]])]]from itertools import chainzipped = (zip(*ele) for ele in zip(*next(zip(*structure))))print (list(chain.from_iterable(zip(*zipped))))[(1, 13), (4, 16), (2, 14), (5, 17), (3, 15), (6, 18)]

Ok a breakdown of the witchcraft:

# transpose sub arrays so column 0 is the first two sub elements from # each sub arrayIn [4]: start = zip(*structure)In [5]: startOut[5]: [(array([[1, 2, 3],         [4, 5, 6]]), array([[13, 14, 15],         [16, 17, 18]])), (array([[ 7,  8,  9],         [10, 11, 12]]), array([[19, 20, 21],         [22, 23, 24]]))]# our interesting sub array's i.e colunm[0]In [6]: first_col = next(start)In [7]: first_colOut[7]: (array([[1, 2, 3],        [4, 5, 6]]), array([[13, 14, 15],        [16, 17, 18]]))# pair up corresponding sub array'sIn [8]: intersting_pairs = zip(*first_col)In [9]: intersting_pairsOut[9]: [(array([1, 2, 3]), array([13, 14, 15])), (array([4, 5, 6]), array([16, 17, 18]))]# pair them up (1, 13), (2, 14) ...In [10]: create_final_pairings = [zip(*ele) for ele in intersting_pairs]In [11]: create_final_pairingsOut[11]: [[(1, 13), (2, 14), (3, 15)], [(4, 16), (5, 17), (6, 18)]]

Finally chain all into a single flat list and get the order correct:

In [13]: from itertools import chain# create flat list In [14]: flat_list = list(chain.from_iterable(zip(*create_final_pairings))In [15]: flat_listOut[15]: [(1, 13), (4, 16), (2, 14), (5, 17), (3, 15), (6, 18)]

A simple example of transposing with zip may help:

In [17]: l = [[1,2,3],[4,5,6]]In [18]: zip(*l)Out[18]: [(1, 4), (2, 5), (3, 6)]In [19]: zip(*l)[0]Out[19]: (1, 4)In [20]: zip(*l)[1]Out[20]: (2, 5)In [21]: zip(*l)[2]Out[21]: (3, 6)

For python2 you can use itertools.izip:

from itertools import chain, izipzipped = (izip(*ele) for ele in izip(*next(izip(*structure))))print (list(chain.from_iterable(izip(*zipped))))[(1, 13), (4, 16), (2, 14), (5, 17), (3, 15), (6, 18)]

I had to write the non-list-comprehension version first to get my head around this:

new_training_vector = []for m1, m2 in zip(structure[0], structure[1]):    for t1, t2 in zip(m1, m2):        for d1, d2 in zip(t1, t2):            new_training_vector.append([d1, d2])

The way it works is by creating two parallel iterators (using zip), one for each model, then creating two parallel iterators for each of the training sets and so on until we get to the actual data and can just stick it together.

Once we have that, it's not hard to go fold it into a list comprehension:

new_training_vector = [[d1, d2]                       for m1, m2 in zip(structure[0], structure[1])                       for t1, t2 in zip(m1, m2)                       for d1, d2 in zip(t1, t2)]

You can also do this with a dictionary, if that works better for some reason. You would lose the order though:

import collectionsd = collections.defaultdict(list)for model in structure:    for i, training_set in enumerate(model):        for j, row in enumerate(training_set):            for k, point in enumerate(row):                d[(i, j, k)].append(point)

The trick to this one is that we just keep track of where we saw each point (except for at the model level), so they automatically go into the same dict item.