'DataFrame' object has no attribute 'withColumn'

Question

I am trying to compare two pandas dataframes but I get an error as 'DataFrame' object has no attribute 'withColumn'. What could be the issue?

import pandas as pd
import pyspark.sql.functions as F

pd_df=pd.DataFrame(df.dtypes,columns=['column','data_type'])
pd_df1=pd.DataFrame(df1.dtypes,columns=['column','data_type'])

pd.merge(pd_df,pd_df1, on='column', how='outer'
    ).withColumn(
    "result",
    F.when(F.col("data_type_x") == 'NaN','new attribute'.otherwise('old attribute')))
    .select(
    "column",
    "data_type_x",
    "data_type_y",
    "result"
    )

df and df1 are some data frames

Answer 1

You mixed up pandas dataframe and Spark dataframe.

The issue is pandas df doesn't have spark function withColumn.