Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

About the dataframe, how to add header to output csv file

Tags:

apache-spark

I have a dataframe like below:

+---+-----+-------------------+
|  x|    y|                  z|
+---+-----+-------------------+
|xyz|12223|A,123@B,456@C,98765|
|abc|12456|              A,123|
+---+-----+-------------------+

I save it as csv:

index.write.csv("D:\\spark\\tmp\\dd2")

two issues here:

  1. Would you like tell me how to save the column name x,y,z to the header of csv please?
  2. How can I save the output to a single file please?
like image 776
Robin Avatar asked Dec 09 '25 03:12

Robin


1 Answers

Would you like tell me how to save the column name x,y,z to the header of csv please?

You have to use header option:

index.write.option("header", "true").write(path)

How can I save the output to a single file please?

Write single CSV file using spark-csv

like image 142
Alper t. Turker Avatar answered Dec 12 '25 11:12

Alper t. Turker



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!