java.io.InvalidClassException: org.apache.spark.deploy.ApplicationDescription; local class incompatible

Question

After locally compiling spark v3.2.1, we deployed it in k8s. We observe below stacktrace in spark-master. Any clues to resolve this exception would be helpful.

• JDK - openjdk8 (1.8)

• Scala - 2.12.15

• Hadoop - 3.3.1

  2022-06-23 05:27:25.847 GMT ERROR [SPARK_MASTER] TransportRequestHandler: [rpc-server-4-1] Error while invoking RpcHandler#receive() for one-way message. java.io.InvalidClassException: org.apache.spark.deploy.ApplicationDescription; local class incompatible: stream classdesc serialVersionUID = 6543101073799644159, local class serialVersionUID = 1574364215946805297

Answer 1

Found out that this exception occurs due to scala version incompatibility. Rebuilding after cleaning up the old spark version image in build machine helped to overcome this issue.