Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why are there two options to read a CSV file in PySpark? Which one should I use?

Spark 2.4.4:

I want to import a CSV file, but there are two options. Why is that? And which one is better? Which one should I use?

from pyspark.sql import SparkSession

spark = SparkSession \
    .builder \
    .master("local[2]") \
    .config('spark.cores.max', '3') \
    .config('spark.executor.memory', '2g') \
    .config('spark.executor.cores', '2') \
    .config('spark.driver.memory','1g') \
    .getOrCreate()

Option 1

df = spark.read \
    .format("com.databricks.spark.csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .load("data/myfile.csv")

Option 2

df = spark.read.load("data/myfile.csv", format="csv", inferSchema="true", header="true")
like image 728
phez1 Avatar asked Oct 29 '25 18:10

phez1


1 Answers

As of Spark 2, com.databricks.spark.csv isn't necessary to write out completely since the CSV reader is included. Therefore option 2 would be preferred.

Or slightly shorter,

spark.read.csv("data/myfile.csv", inferSchema=True, header=True)

But option 2 would be better if you extracted the input format into some configuration file

like image 200
OneCricketeer Avatar answered Nov 01 '25 08:11

OneCricketeer



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!