Pandas drop_duplicates method not working on dataframe containing lists

drop_duplicates won't work with lists in your dataframe as the error message implies. However, you can drop duplicates on the dataframe casted as str and then extract the rows from original df using the index from the results.

Setup

df = pd.DataFrame({'Keyword': {0: 'apply', 1: 'apply', 2: 'apply', 3: 'terms', 4: 'terms'}, 'X': {0: [1, 2], 1: [1, 2], 2: 'xy', 3: 'xx', 4: 'yy'}, 'Y': {0: 'yy', 1: 'yy', 2: 'yx', 3: 'ix', 4: 'xi'}})#Drop directly causes the same errordf.drop_duplicates()Traceback (most recent call last):...TypeError: unhashable type: 'list'

Solution

#convert hte df to str type, drop duplicates and then select the rows from original df.df.loc[df.astype(str).drop_duplicates().index]Out[205]:   Keyword       X   Y0   apply  [1, 2]  yy2   apply      xy  yx3   terms      xx  ix4   terms      yy  xi#the list elements are still list in the final results.df.loc[df.astype(str).drop_duplicates().index].loc[0,'X']Out[207]: [1, 2]

Edit: replaced iloc with loc. In this particular case, both work as the index matches the positional index, but it is not general

@Allen's answer is great, but have a little problem.

df.iloc[df.astype(str).drop_duplicates().index]

it should be loc not iloc.loot at the example.

a = pd.DataFrame([['a',18],['b',11],['a',18]],index=[4,6,8])Out[52]:    0   14  a  186  b  118  a  18a.iloc[a.astype(str).drop_duplicates().index]Out[53]:...IndexError: positional indexers are out-of-boundsa.loc[a.astype(str).drop_duplicates().index]Out[54]:    0   14  a  186  b  11

Overview: you can see which rows are duplicated

Method 1:

df2=df.copy()mylist=df2.iloc[0,1]df2.iloc[0,1]=' '.join(map(str,mylist))mylist=df2.iloc[1,1]df2.iloc[1,1]=' '.join(map(str,mylist))duplicates=df2.duplicated(keep=False)print(df2[duplicates])

Method 2:

print(df.astype(str).duplicated(keep=False))