How to convert DataFrame to Json?

Question

I have a huge JSON file, a small part from it as follows:

{
    "socialNews": [{
        "adminTagIds": "",
        "fileIds": "",
        "departmentTagIds": "",
        ........
        ........
        "comments": [{
            "commentId": "",
            "newsId": "",
            "entityId": "",
            ....
            ....
        }]
    }]
    .....
    }

I have applied lateral view explode on socialNews as follows:

val rdd = sqlContext.jsonFile("file:///home/ashish/test")
rdd.registerTempTable("social")
val result = sqlContext.sql("select * from social LATERAL VIEW explode(socialNews) social AS comment")

Now I want to convert back this result (DataFrame) to JSON and save into a file, but I am not able to find any Scala API to do the conversion. Is there any standard library to do this or some way to figure it out?

Answer 1

val result: DataFrame = sqlContext.read.json(path)
result.write.json("/yourPath")

The method write is in the class DataFrameWriter and should be accessible to you on DataFrame objects. Just make sure that your rdd is of type DataFrame and not of deprecated type SchemaRdd. You can explicitly provide type definition val data: DataFrame or cast to dataFrame with toDF().