Convert a list of dictionary to a a dataframe

Question

I have a list of dictionary which looks like:

list_dict = [{'test1':{'a':1,'b':12,'c':40,'d':120,'e':20,'f':1,'g':2,'h':'2'}},
                  {'test2':{'a':5,'b':'10','c':20}},
                   {'test3':{'e':21,'f':'18','g':22,'h':20}}]

I would want to convert this to a dataframe like this: The keys should come as rows and tests should come as columns. and where the test doesn't have a key which is there in other tests should be filled with values as NAN

    mac_type  test1  test2  test3
    a         1      5      NAN
    b         12     10     NAN
    c         40     20     NAN
    d         120    NAN    NAN
    e         20     NAN    21
    f         1      NAN    18
    g         2      NAN    22
    h         2      NAN    20

Please help me in this.

Answer 1

Use dict comprehension with flatten nested dicts and pass to Dataframe constructor:

df = pd.DataFrame({k: v for x in list_dict for k, v in x.items()})
print (df)
  test1 test2 test3
a     1     5   NaN
b    12    10   NaN
c    40    20   NaN
d   120   NaN   NaN
e    20   NaN    21
f     1   NaN    18
g     2   NaN    22
h     2   NaN    20

Or create DataFrame for each nested dictionary and pass to concat, if large dictionaries and many of outer keys this should be slowier like first solution:

df = pd.concat([pd.DataFrame(x) for x in list_dict], axis=1)
print (df)
  test1 test2 test3
a     1     5   NaN
b    12    10   NaN
c    40    20   NaN
d   120   NaN   NaN
e    20   NaN    21
f     1   NaN    18
g     2   NaN    22
h     2   NaN    20

Answer 2

Use reduce before build your DataFrame

from functools import reduce
df = pd.DataFrame(reduce(lambda cum_dict, new_dict: dict(cum_dict, **new_dict), 
                         list_dict))

print (df)
  test1 test2 test3
a     1     5   NaN
b    12    10   NaN
c    40    20   NaN
d   120   NaN   NaN
e    20   NaN    21
f     1   NaN    18
g     2   NaN    22
h     2   NaN    20