DataFrame error: “overloaded method value select with alternatives”

Question

I tried to create a new dataframe by select hour+minute/60 and other columns from a dataframe as follows:

val logon11 = logon1.select("User","PC","Year","Month","Day","Hour","Minute",$"Hour"+$"Minute"/60)

I got the error below:

<console>:38: error: overloaded method value select with alternatives:
  (col: String,cols: String*)org.apache.spark.sql.DataFrame <and>
  (cols: org.apache.spark.sql.Column*)org.apache.spark.sql.DataFrame
cannot be applied to (String, String, String, String, String, String, String,org.apache.spark.sql.Colum)

...

Maybe I have known the reason is that I cannot get a DataFrame with these types using "select" at the same time. Then how can I get such dataframe?

Answer 1

DF's select method takes arguments of type either all Strings or all org.apache.spark.sql.Columns but doesn't take mix of both.

In your case you are passing both String and Column type parameters to select method.

val logon11 = logon1.select($"User",$"PC",$"Year",$"Month",$"Day",$"Hour",$"Minute",$"Hour"+$"Minute"/60 as "total_hours")

Hope it helps!