Thursday, June 20, 2013

Python - Remove duplicate dictionaries in a list

Assuming I have a list of dictionaries, and I want to remove all of duplicate item from that list. The set built-in type of Python is what we need:


def remove_duplicate_items(my_list):
return_list = []
myset = set()
for i in my_list:
t = tuple(i.items())
if t not in myset:
myset.add(t)
return_list.append(i)

return return_list

my_list = [{'a': 1, 'b': 2}, {'a': 4, 'b': 7}, {'a': 1, 'b': 2}]

my_final_list = remove_duplicate_items(my_list)

>>> [{'a': 1, 'b': 2}, {'a': 4, 'b': 7}]

Remember: a set only accepts tuples as its items.