JAVA_HOME is set but does not point to a valid version

Question

I'm attempting to use sparklyr to analyze a large dataset in R. Upon my attempts to establish a Spark connection with spark_connect, I receive the following error:

Error in get_java(throws = TRUE) : Java is required to connect to Spark. JAVA_HOME is set but does not point to a valid version. Please fix JAVA_HOME or reinstall from: https://www.java.com/en/

I've reinstalled Java but continue to get the same error. Any advice?

Answer 1

When I run:

sparklyr:::get_java()
           java 
"/usr/bin/java" 

It appears that you don't have java set up in such a manner that the response for that sparklyr function is satisfactory. Unlike @Kerie I get nothing from the echo command. Instead, I can get sensible results from this command in a Terminal session:

$ java -version
#-------------------
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

Running MacOS 10.11.6 (not upgraded because my hardware is "obsolete" per Apple) and R 3.5.1.

There's an irony here in that it appears that the get_java function is supposed to set a value for the location if it cannot find an environment variable. Here's the code:

sparklyr:::get_java
 #----------
function (throws = FALSE) 
{
    java_home <- Sys.getenv("JAVA_HOME", unset = NA)
    if (!is.na(java_home)) {
        java <- file.path(java_home, "bin", "java")
        if (identical(.Platform$OS.type, "windows")) {
            java <- paste0(java, ".exe")
        }
        if (!file.exists(java)) {
            if (throws) {
                stop("Java is required to connect to Spark. ", 
                  "JAVA_HOME is set but does not point to a valid version. ", 
                  "Please fix JAVA_HOME or reinstall from: ", 
                  java_install_url())
            }
            java <- ""
        }
    }
    else java <- Sys.which("java")
    java
}
<bytecode: 0x7fb5c7f2db30>
<environment: namespace:sparklyr>

Because I do not have an environment variable for JAVA_HOME, but do have java registered with which, the get_java function returns a valid path. So my system returns:

Sys.which("java")
           java 
"/usr/bin/java" 

From comments by @user6910411, I am reminded that you should not update to the current Java Dev Kit (which is 1.9), but do rather use the link provided by @Kerie to the prior major version, 1.8. You should also run:

Sys.unsetenv("JAVA_HOME") 

to get rid of the misleading symlink. Or perhaps you could track it down at /Library/Java/Home (if that's where it is) and delete it before installing the newer (but not newest) version.