Replace missing values in list from second list using python/pandas

Question

Consider you have two lists (or columns in a pandas DataFrame), each containing some null values. You want a single list that replaces the null values in one list with the corresponding non-null values of the other if one exists.

Example:

s1 = [1, NaN, NaN]
s2 = [NaN, NaN, 3]
## some function
result = [1, NaN, 3]

Assume that if both lists are non-null at some position then they match, so we need not worry about resolving conflicts. If so, I know I can solve it with a list comprehension:

[x if ~np.isnan(x) else y for (x,y) in zip(s1,s2)]

or if s1 and s2 are columns in a pandas DataFrame df, then we can use similar logic and the apply function:

df.apply(lambda x: x.s1 if ~np.isnan(x.s1) else x.s2, axis=1)

but is there a cleaner way to do this, perhaps using some of the pandas functionality? What is this kind of operation even called? It is kind of like a union, but preserves ordering and null values when lacking an alternative.

Answer 1

You can use pandas fillna functionality to fill missing values from other columns.

df = pd.DataFrame([[1,np.nan],[np.nan,np.nan],[np.nan,3]],columns=['c1','c2'])
df['c1'].fillna(df['c2'])