Is there a way to find out which port the Spark web UI is using?

Question

I start my Spark session in Jupyter with something like this:

from pyspark.sql import SparkSession

spark = ( SparkSession
            .builder
            .appName('analysis')
            .getOrCreate() )

Is there a way to find out which port the web UI is using? Thanks!

Answer 1

You can use uiWebUrl property:

>>> spark.sparkContext.uiWebUrl
'http://some_host:4040'

and for example split

>>> spark.sparkContext.uiWebUrl.split(":")[-1]
'4040'

Answer 2

This is how I figured it out in pyspark.

>>> import pyspark
>>> sc = pyspark.SparkContext(appName="some_app")
>>> sc.uiWebUrl

[1]: http://localhost:4041/jobs/