Pyspark error with UDF: py4j.Py4JException: Method __getnewargs__([]) does not exist error

Question

I am trying to solve the following error (I am using the databricks platform and spark 2.0)

tweets_cleaned.createOrReplaceTempView("tweets_cleanedSQL")
def Occ(keyword):
  occurences = spark.sql("SELECT * \
                                FROM tweets_cleanedSQL \
                                WHERE LOWER(text) LIKE '%" + keyword + "%' \
                            ")
  return occurences.count()


occurences_udf = udf(Occ)

If I run this code, I receive the following error:

py4j.Py4JException: Method getnewargs([]) does not exist ==> error only occurs when trying to define the udf.

Answer 1

the UDF function turns a regular function to a function that is applied on any element of the input column. You cannot have this function call spark functions (in this case you are calling spark.sql which would in turn need to create workers etc. which is not supported.