Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How register UDF without arguments in Apache Spark by Java

I need to register udf function without arguments. But Apache Spark have not UDF0 interface realization. I trying somethig like:

UDF1<Object, String> my_func = o -> return "some_generated_string";
sqlContext.udf().register("my_func", my_func, DataTypes.StringType);

But df.withColumns("newCol", functions.expr("concat(col1, my_funct())")); returns exception org.apache.spark.sql.UDFRegistration$$anonfun$register$25$$anonfun$apply$1 cannot be cast to scala.Function0.

So df.withColumns("newCol", functions.expr("concat(col1, my_funct(1))")); work correctly but this is wrong way and smells bad.

UDFRegistration in org.apache.spark.sql has method register[RT: TypeTag](name: String, func: Function0[RT]): UserDefinedFunction. Java see this method as register(String name, Function0<RT> func, TypeTag<RT> evidence$1). I can wrote scala.Function0 implementation, but what is TypeTag evidence$1?

like image 272
Andrei Iatsuk Avatar asked Sep 16 '26 04:09

Andrei Iatsuk


1 Answers

I resolve this problem by next trick:

UDF1<Object, String> my_func = o -> "some_generated_string";
sqlContext.udf().register("my_func", my_func, DataTypes.StringType);

String expression = "concat(`col1`, my_func())";
expression = expression.replace("my_func()", "my_func(null)");

df.withColumns("newCol", functions.expr(expression));
like image 64
Andrei Iatsuk Avatar answered Sep 19 '26 03:09

Andrei Iatsuk



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!