Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to create Dataset (not DataFrame) without using case class but using StructType?

How can I create Dataset using StructType?

We can create a Dataset as follows:

case class Person(name: String, age: Int)

val personDS = Seq(Person("Max", 33), Person("Adam", 32), Person("Muller", 
62)).toDS()
personDS.show()

Is there a way to create a Dataset without using a case class?

I'd like to create a DataFrame using a case class and using StructType.

like image 631
Narendra Parmar Avatar asked Sep 06 '25 20:09

Narendra Parmar


1 Answers

If you know how to create DataFrame, you already now how to create Dataset :)

DataFrame = Dataset[Row].

What it means? Try:

val df : DataFrame = spark.createDataFrame(...) // with StructType
import org.apache.spark.sql._
val ds : Dataset[Row] = df; // no error, as DataFrame is only a type alias of Dataset[Row]
like image 179
T. Gawęda Avatar answered Sep 09 '25 22:09

T. Gawęda