Hadoop Configuration in Spark

Question

I need to get hold of the current Hadoop org.apache.hadoop.conf.Configuration in my Spark job, for debugging purposes. Specifically I need to get org.apache.hadoop.fs.FileSystem for a path with the org.apache.hadoop.fs.Path#getFileSystem(conf: Configuration) method

Given a org.apache.spark.SparkContext, is there a way to get the Configuration?

Answer 1

you can set configration as per below code

sc.hadoopConfiguration.set("my.mapreduce.setting","someValue")