What is the most time efficient way to remove unordered duplicates in a 2D array?

Since you want to find unordered duplicates the best way to go is by typecasting. Typecast them as set. Since set only contains immutable elements. So, I made a set of tuples.

Note: The best way to eliminate duplicates is by making a set of the given elements.

>>> set(map(tuple,map(sorted,x))){(-3, -2, 4, 5), (-5, 0, 4, 5)}

The best way is to not generate the duplicates in the first place.

The idea is to first create all possible combinations of values that appear multiple times, where each appears 0, 1, ... times. Then, we complete them with all possible combinations of the unique elements.

from itertools import combinations, product, chainfrom collections import Counternums = [-5,5,4,-3,0,0,4,-2]def combinations_without_duplicates(nums, k):    counts = Counter(nums)    multiples = {val: count for val, count in counts.items() if count >= 2 }    uniques = set(counts) - set(multiples)                  possible_multiples = [[[val]*i for i in range(count+1)] for val, count in multiples.items()]    multiples_part = (list(chain(*x)) for x in product(*possible_multiples))    # omit the ones that are too long    multiples_part = (lst for lst in multiples_part if len(lst) <= k)    # Would be at this point:    # [[], [0], [0, 0], [4], [4, 0], [4, 0, 0], [4, 4], [4, 4, 0], [4, 4, 0, 0]]    for m_part in multiples_part:        missing = k - len(m_part)        for c in combinations(uniques, missing):            yield m_part + list(c)list(combinations_without_duplicates(nums, 4))

Output:

[[-3, -5, 5, -2], [0, -3, -5, 5], [0, -3, -5, -2], [0, -3, 5, -2], [0, -5, 5, -2], [0, 0, -3, -5], [0, 0, -3, 5], [0, 0, -3, -2], [0, 0, -5, 5], [0, 0, -5, -2], [0, 0, 5, -2], [4, -3, -5, 5], [4, -3, -5, -2], [4, -3, 5, -2], [4, -5, 5, -2], [4, 0, -3, -5], [4, 0, -3, 5], [4, 0, -3, -2], [4, 0, -5, 5], [4, 0, -5, -2], [4, 0, 5, -2], [4, 0, 0, -3], [4, 0, 0, -5], [4, 0, 0, 5], [4, 0, 0, -2], [4, 4, -3, -5], [4, 4, -3, 5], [4, 4, -3, -2], [4, 4, -5, 5], [4, 4, -5, -2], [4, 4, 5, -2], [4, 4, 0, -3], [4, 4, 0, -5], [4, 4, 0, 5], [4, 4, 0, -2], [4, 4, 0, 0]]

You can use itertools.combinations_with_replacement().

Return r length subsequences of elements from the input iterable allowing individual elements to be repeated more than once.

Combinations are emitted in lexicographic sort order. So, if the input iterable is sorted, the combination tuples will be produced in sorted order.

Elements are treated as unique based on their position, not on their value. So if the input elements are unique, the generated combinations will also be unique.

Source: Python Docs