NoClassDefFoundError for joda DateTimeFormat

Question

I am trying to use joda's DateTimeFormat in a Spark project (in Scala) for which I am using sbt as a build tool, and it is giving me a java.lang.NoClassDefFoundError when I try to run it as a standalone application. This seems odd to me because I don't get any errors when I play with it in the console, but when I try to run it as an application, it errors with

Exception in thread "main" java.lang.NoClassDefFoundError: org/joda/time/format/DateTimeFormat
    at com.bdcoe.poc.spark.analytics.Driver$.main(Driver.scala:35)
    at com.bdcoe.poc.spark.analytics.Driver.main(Driver.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.joda.time.format.DateTimeFormat
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    ... 9 more

Here is my build.sbt file, which is located in the project's root directory:

import AssemblyKeys._
import net.virtualvoid.sbt.graph.Plugin.graphSettings

organization := "com.bdcoe.poc"

name := "spark-analytics"

scalaVersion := "2.10.4"

resolvers ++= Seq(
  "Local Maven" at Path.userHome.asFile.toURI.toURL + ".m2/repository"
)

libraryDependencies ++= {
  val slf4jVersion = "1.7.7"
  Seq(
    "org.apache.spark" %% "spark-core" % "1.0.0",
    "com.github.nscala-time" %% "nscala-time" % "1.2.0",
    "org.scalatest" %% "scalatest" % "2.0" % "test",
    "org.slf4j" % "slf4j-api" % slf4jVersion % "provided",
    "org.slf4j" % "slf4j-nop" % slf4jVersion % "test",
    "joda-time" % "joda-time" % "2.2", 
    "org.joda" % "joda-convert" % "1.2"
  )
}

And here is my import/usage of joda:

import org.joda.time.format.DateTimeFormat.{ forPattern => formatFor }

val dateTime = formatFor("YYYY-MM-dd HH:mm").parseDateTime _ 
val timeStamp: DateTime = dateTime(args(2))

Any suggestions on why it can find joda in the sbt console but not when run as a standalone application?

Answer 1

The problem is with the classpath during runtime.

I've noticed you are using sbt-assembly. Add this assemblySettings line after imports in your build.sbt:

import AssemblyKeys._
import net.virtualvoid.sbt.graph.Plugin.graphSettings

assemblySettings

organization := "com.bdcoe.poc"

...

as described in the docs: https://github.com/sbt/sbt-assembly. Sbt-assembly will find the main class and package the JAR with a properly configured manifest.

Note that the task you should run is assembly, not package. Without assemblySettings included in your project you would not be able to run this task previously, and since you were running your packaged JAR (most likely) then you've must have generated it with package task, and not with assembly.