What's the difference between Sparkconf and Sparkcontext?

Question

I've encountered a problem with pyspark when I've made Import Pyspark from Sparkcontext but I found that it can be imported from sparkconf as well, I'm asking what's the difference between those two spark class libraries.

Answer 1

Sparkcontext is the entry point for spark environment. For every sparkapp you need to create the sparkcontext object. In spark 2 you can use sparksession instead of sparkcontext.

Sparkconf is the class which gives you the various option to provide configuration parameters.

Val Conf = new sparkConf().setMaster(“local[*]”).setAppName(“test”)
Val SC  = new sparkContext(Conf)

The spark configuration is passed to spark context. You can also set different application configuration in sparkconf and pass to sparkcontex