Get unique values from a list in python [duplicate] Get unique values from a list in python [duplicate] python python

# Get unique values from a list in python [duplicate]

First declare your list properly, separated by commas. You can get the unique values by converting the list to a set.

``mylist = ['nowplaying', 'PBS', 'PBS', 'nowplaying', 'job', 'debate', 'thenandnow']myset = set(mylist)print(myset)``

If you use it further as a list, you should convert it back to a list by doing:

``mynewlist = list(myset)``

Another possibility, probably faster would be to use a set from the beginning, instead of a list. Then your code should be:

``output = set()for x in trends:    output.add(x)print(output)``

As it has been pointed out, sets do not maintain the original order. If you need that, you should look for an ordered set implementation (see this question for more).

To be consistent with the type I would use:

``mylist = list(set(mylist))``

``used = set()mylist = [u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']unique = [x for x in mylist if x not in used and (used.add(x) or True)]``

And one more solution using `reduce` and without the temporary `used` var.

``mylist = [u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']unique = reduce(lambda l, x: l.append(x) or l if x not in l else l, mylist, [])``

UPDATE - Dec, 2020 - Maybe the best approach!

Starting from python 3.7, the standard dict preserves insertion order.

Changed in version 3.7: Dictionary order is guaranteed to be insertion order. This behavior was an implementation detail of CPython from 3.6.

So this gives us the ability to use `dict.from_keys` for de-duplication!

NOTE: Credits goes to @rlat for giving us this approach in the comments!

``mylist = [u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']unique = list(dict.fromkeys(mylist))``

In terms of speed - for me its fast enough and readable enough to become my new favorite approach!

UPDATE - March, 2019

And a 3rd solution, which is a neat one, but kind of slow since `.index` is O(n).

``mylist = [u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']unique = [x for i, x in enumerate(mylist) if i == mylist.index(x)]``

UPDATE - Oct, 2016

Another solution with `reduce`, but this time without `.append` which makes it more human readable and easier to understand.

``mylist = [u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']unique = reduce(lambda l, x: l+[x] if x not in l else l, mylist, [])#which can also be writed as:unique = reduce(lambda l, x: l if x in l else l+[x], mylist, [])``

NOTE: Have in mind that more human-readable we get, more unperformant the script is. Except only for the `dict.from_keys` approach which is python 3.7+ specific.

``import timeitsetup = "mylist = [u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']"#10x to Michael for pointing out that we can get faster with set()timeit.timeit('[x for x in mylist if x not in used and (used.add(x) or True)]', setup='used = set();'+setup)0.2029558869980974timeit.timeit('[x for x in mylist if x not in used and (used.append(x) or True)]', setup='used = [];'+setup)0.28999493700030143# 10x to rlat for suggesting this approach!   timeit.timeit('list(dict.fromkeys(mylist))', setup=setup)0.31227896199925453timeit.timeit('reduce(lambda l, x: l.append(x) or l if x not in l else l, mylist, [])', setup='from functools import reduce;'+setup)0.7149233570016804timeit.timeit('reduce(lambda l, x: l+[x] if x not in l else l, mylist, [])', setup='from functools import reduce;'+setup)0.7379565160008497timeit.timeit('reduce(lambda l, x: l if x in l else l+[x], mylist, [])', setup='from functools import reduce;'+setup)0.7400134069976048timeit.timeit('[x for i, x in enumerate(mylist) if i == mylist.index(x)]', setup=setup)0.9154880290006986``

Because @monica asked a good question about "how is this working?". For everyone having problems figuring it out. I will try to give a more deep explanation about how this works and what sorcery is happening here ;)

I try to understand why `unique = [used.append(x) for x in mylist if x not in used]` is not working.

Well it's actually working

``>>> used = []>>> mylist = [u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']>>> unique = [used.append(x) for x in mylist if x not in used]>>> print used[u'nowplaying', u'PBS', u'job', u'debate', u'thenandnow']>>> print unique[None, None, None, None, None]``

The problem is that we are just not getting the desired results inside the `unique` variable, but only inside the `used` variable. This is because during the list comprehension `.append` modifies the `used` variable and returns `None`.

So in order to get the results into the `unique` variable, and still use the same logic with `.append(x) if x not in used`, we need to move this `.append` call on the right side of the list comprehension and just return `x` on the left side.

But if we are too naive and just go with:

``>>> unique = [x for x in mylist if x not in used and used.append(x)]>>> print unique[]``

We will get nothing in return.

Again, this is because the `.append` method returns `None`, and it this gives on our logical expression the following look:

``x not in used and None``

This will basically always:

1. evaluates to `False` when `x` is in `used`,
2. evaluates to `None` when `x` is not in `used`.

And in both cases (`False`/`None`), this will be treated as `falsy` value and we will get an empty list as a result.

But why this evaluates to `None` when `x` is not in `used`? Someone may ask.

Well it's because this is how Python's short-circuit operators works.

The expression `x and y` first evaluates x; if x is false, its value isreturned; otherwise, y is evaluated and the resulting value isreturned.

So when `x` is not in used (i.e. when its `True`) the next part or the expression will be evaluated (`used.append(x)`) and its value (`None`) will be returned.

But that's what we want in order to get the unique elements from a list with duplicates, we want to `.append` them into a new list only when we they came across for a fist time.

So we really want to evaluate `used.append(x)` only when `x` is not in `used`, maybe if there is a way to turn this `None` value into a `truthy` one we will be fine, right?

Well, yes and here is where the 2nd type of `short-circuit` operators come to play.

The expression `x or y` first evaluates x; if x is true, its value isreturned; otherwise, y is evaluated and the resulting value isreturned.

We know that `.append(x)` will always be `falsy`, so if we just add one `or` next to him, we will always get the next part. That's why we write:

``x not in used and (used.append(x) or True)``

so we can evaluate `used.append(x)` and get `True` as a result, only when the first part of the expression `(x not in used)` is `True`.

Similar fashion can be seen in the 2nd approach with the `reduce` method.

``(l.append(x) or l) if x not in l else l#similar as the above, but maybe more readable#we return l unchanged when x is in l#we append x to l and return l when x is not in ll if x in l else (l.append(x) or l)``

where we:

1. Append `x` to `l` and return that `l` when `x` is not in `l`. Thanks to the `or` statement `.append` is evaluated and `l` is returned after that.
2. Return `l` untouched when `x` is in `l`