Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python Pandas -- why does the `in` operator work with indices and not with the data?

Tags:

python

pandas

I discovered the hard way that Pandas in operator, applied to Series operates on indices and not on the actual data:

In [1]: import pandas as pd

In [2]: x = pd.Series([1, 2, 3])

In [3]: x.index = [10, 20, 30]

In [4]: x
Out[4]:
10    1
20    2
30    3
dtype: int64

In [5]: 1 in x
Out[5]: False


In [6]: 10 in x
Out[6]: True

My intuition is that x series contains the number 1 and not the index 10, which is apparently wrong. What is the reason behind this behavior? Are the following approaches the best possible alternatives?

In [7]: 1 in set(x)
Out[7]: True

In [8]: 1 in list(x)
Out[8]: True

In [9]: 1 in x.values
Out[9]: True

UPDATE

I did some timings on my suggestions. It looks like x.values is the best way:

In [21]: x = pd.Series(np.random.randint(0, 100000, 1000))

In [22]: x.index = np.arange(900000, 900000 + 1000)

In [23]: x.tail()
Out[23]:
900995    88999
900996    13151
900997    25928
900998    36149
900999    97983
dtype: int64

In [24]: %timeit 36149 in set(x)
10000 loops, best of 3: 190 µs per loop

In [25]: %timeit 36149 in list(x)
1000 loops, best of 3: 638 µs per loop

In [26]: %timeit 36149 in (x.values)
100000 loops, best of 3: 6.86 µs per loop
like image 853
Boris Gorelik Avatar asked Jan 29 '26 18:01

Boris Gorelik


1 Answers

It is may be helpful to think of the pandas.Series as being a bit like a dictionary, where the index values are equivalent to the keys. Compare:

>>> d = {'a': 1}
>>> 1 in d
False
>>> 'a' in d
True

with:

>>> s = pandas.Series([1], index=['a'])
>>> 1 in s
False
>>> 'a' in s
True

However, note that iterating over the series iterates over the data, not the index, so list(s) would give [1], not ['a'].

Indeed, per the documentation, the index values "must be unique and hashable", so I'd guess there's a hashtable under there somewhere.

like image 173
jonrsharpe Avatar answered Jan 31 '26 08:01

jonrsharpe



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!