How to extract rows in a pandas dataframe NOT in a subset dataframe

Question

I have two dataframes. DF and SubDF. SubDF is a subset of DF. I want to extract the rows in DF that are NOT in SubDF.

I tried the following:

DF2 = DF[~DF.isin(SubDF)]

The number of rows are correct and most rows are correct,

ie number of rows in subDF + number of rows in DF2 = number of rows in DF

but I get rows with NaN values that do not exist in the original DF

Not sure what I'm doing wrong.

Note: the original DF does not have any NaN values, and to double check I did DF.dropna() before and the result still produced NaN

Answer 1

You need merge with outer join and boolean indexing, because DataFrame.isin need values and index match:

DF = pd.DataFrame({'A':[1,2,3],
                   'B':[4,5,6],
                   'C':[7,8,9],
                   'D':[1,3,5],
                   'E':[5,3,6],
                   'F':[7,4,3]})

print (DF)
   A  B  C  D  E  F
0  1  4  7  1  5  7
1  2  5  8  3  3  4
2  3  6  9  5  6  3

SubDF = pd.DataFrame({'A':[3],
                   'B':[6],
                   'C':[9],
                   'D':[5],
                   'E':[6],
                   'F':[3]})

print (SubDF)
   A  B  C  D  E  F
0  3  6  9  5  6  3

#return no match
DF2 = DF[~DF.isin(SubDF)]
print (DF2)
   A  B  C  D  E  F
0  1  4  7  1  5  7
1  2  5  8  3  3  4
2  3  6  9  5  6  3

DF2 = pd.merge(DF, SubDF, how='outer', indicator=True)
DF2 = DF2[DF2._merge == 'left_only'].drop('_merge', axis=1)
print (DF2)
   A  B  C  D  E  F
0  1  4  7  1  5  7
1  2  5  8  3  3  4

Answer 2

Another way, borrowing the setup from @jezrael:

df = pd.DataFrame({'A':[1,2,3],
                   'B':[4,5,6],
                   'C':[7,8,9],
                   'D':[1,3,5],
                   'E':[5,3,6],
                   'F':[7,4,3]})

sub = pd.DataFrame({'A':[3],
                    'B':[6],
                    'C':[9],
                    'D':[5],
                    'E':[6],
                    'F':[3]})

extract_idx = list(set(df.index) - set(sub.index))
df_extract = df.loc[extract_idx]

The rows may not be sorted in the original df order. If matching order is required:

extract_idx = list(set(df.index) - set(sub.index))
idx_dict = dict(enumerate(df.index))
order_dict = dict(zip(idx_dict.values(), idx_dict.keys()))
df_extract = df.loc[sorted(extract_idx, key=order_dict.get)]