spark: WARN amfilter.AmIpFilter: Could not find proxy-user cookie, so user will not be set

Question

Do you know why this error below happens in spark shell when I try to access spark UI master:4040?

 WARN amfilter.AmIpFilter: Could not find proxy-user cookie, so user will not be set

Answer 1

This happens, if you start spark shell with yarn.

spark-shell --master yarn

In that case, YARN will start a proxy web application to increase the security of the overall system. The URL of the proxy will be displayed in the log, while starting the Spark shell. Here is a sample from my log:

16/06/26 08:38:28 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> myhostname, PROXY_URI_BASES -> http://myhostname:8088/proxy/application_1466917030969_0003), /proxy/application_1466917030969_0003

You shouldn't access the standard Spark Web UI using port 4040 (or whatever you have configured). Instead I know these 2 options (where I prefere the 2nd one):

1. Scan the log for the proxy application URL and use that

2. Open the YARN Web UI http://localhost:8088/cluster and follow the link to the ApplicationMaster (column Tracking UI) of the running Spark application

This is also descibed briefly in the YARN and SPark documentation.

Spark Security documentation:
https://spark.apache.org/docs/latest/security.html

Yarn Web Application Proxy documentation:
https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html