Python - remove duplicate of dict that contains list of lists

Question

suppose training_data is a dict that contains a list of lists such as {1: [[1, 2], [1, 3], [1, 2, 5]], 2: [[1], [5], [1, 6]], 3: [[7], [5]]}

I want to merge each value in training_data and then flatten it once to a list so that it will become [[1,2,3,5],[1,5,6],[7,5]].

Answer 1

you can achieve the same result with a one-line comprehension, chaining and using a set + sort

import itertools

d = {1: [[1, 2], [1, 3], [1, 2, 5]], 2: [[1], [5], [1, 6]], 3: [[7], [5]]}

result = [sorted(set(itertools.chain.from_iterable(v))) for v in d.values()]

print(result)

outputs:

[[1, 2, 3, 5], [1, 5, 6], [5, 7]]

Answer 2

If you only need unique elements from the list without maintaining the order of elements in initial list, then you may use set here with itertools.chain.from_iterable. You may use these with list comprehension to achieve your desired list as:

>>> from itertools import chain
>>> my_dict = {1: [[1, 2], [1, 3], [1, 2, 5]], 2: [[1], [5], [1, 6]], 3: [[7], [5]]}

>>> [list(set(chain.from_iterable(v))) for v in my_dict.values()]
[[1, 2, 3, 5], [1, 5, 6], [5, 7]]

Note: since dictionaries are unordered in Python <3.6, the order of sub-lists in the resultant lists might differ.

For preserving the order of elements in the sublist while merging, take a look at: Pythonic way to merge two overlapping lists, preserving order