Find duplicate values in two arrays, Python

Question

I have two arrays (A and B) with about 50 000 values in each. Every value represents an ID. I want to create a pandas dataframe with three columns, col1: values from array A, col2: values from array B, col3: a string with the labels "unique" or "duplicate". In each array the ID:s are unique.

The arrays is of different length. So I can't do something like this to get started.

a = np.array([1, 2, 3, 4, 5])
a = np.array([5, 6, 7, 8, 9, 10])
pd.DataFrame({'a':a, 'a':b})

I was then thinking to create a different pandas dataframe, also with three columns. One for ID, another for which array the ID comes from (a or b). And thereafter group on ID and count occurrences. if >=2 then we have a duplicate.

But I couldn’t figure out how get to numpy arrays after one another in the same column (like rbind in R) and at the same time create the other column based on which array the value come from.

Most likely there are far better solutions that those I have suggested above. Any ideas?

Answer 1

For finding duplicate elements in two arrays, use numpy.intersect1d:

In [458]: a = np.array([1, 2, 3, 4, 5])

In [459]: b = np.array([5, 6, 7, 8, 9, 10])

In [462]: np.intersect1d(a,b)
Out[462]: array([5])

Answer 2

Convert the array into series and then concat them to create dataframe

a = np.array([1, 2, 3, 4, 5,])
b = np.array([5, 6, 7, 8, 9, 10])

s1 = pd.Series(a, name = 'a')
s2 = pd.Series(b, name = 'b')
pd.concat([s1, s2], axis = 1)

     a  b
0   1.0 5
1   2.0 6
2   3.0 7
3   4.0 8
4   5.0 9
5   NaN 10