AWS EMR cluster with Flink does not run any Jar, instead gives java.lang.NoSuchMethodError

Question

I'm trying to submit a jar as a step in AWS EMR using the following command:

aws emr add-steps --cluster-id j-XXXXXXXXX --steps Type=CUSTOM_JAR,Name=test-job,ActionOnFailure=CONTINUE,Jar=s3://xxxx/frauddetection-0.1.jar,Args=flink-yarn-session,-d

It is a basic EMR cluster with Flink 1.14.2 and JDK 8 Adding this as a step gave the error:

java.lang.NoSuchMethodError:org.apache.commons.math3.stat.descriptive.rank.Percentile.withNaNStrategy

I checked the Jar and apache commoms library is present in it. JDK and Flink version on my local machine are the same as that on EMR and the jar runs without any issue locally

Answer 1

One possibility is that a different (typically older) version of the Apache Commons Math3 jar is part of the AWS EMR installation, and so that's what is getting picked up first on the classpath.

You could try shading those classes when you build your jar.