Java Spring Boot app with multiple database drivers doesn't run on Linux

Question

This question is NOT about setting up multiple datasources.

We are migrating from database A (Type: Sybase) to database B (Type: MSSQL) but temporarily need to support both while this migration is ongoing. The tables/schema in these two databases are identical. In a Java 11 spring Boot 2.X application, I want to be flexible in which database to use when deploying the application by implementing all datasource configuration in two distinct profiles. This works fine when running locally (IntelliJ/Windows). Running the maven clean install both inside and outside of IntelliJ works fine.

For unit tests, we use dbunit, with an H2 in memory database. This requires an addtional driver. All unit tests pass successfully locally (IntelliJ or Maven/Windows). When running the unit tests, the application's main configuration is not used and all datasource configuration is 100% dbunit configuration.

So everything looks in order right? Unfortunately, when our Jenkins build server (Linux!) runs the dbunit unittest, the HikariPool can't properly start for those test that require it because of this rather cryptic error:

2020-03-26 17:56:27.366  INFO 15845 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-7 - Starting...
2020-03-26 17:56:27.369  WARN 15845 --- [           main] o.s.w.c.s.GenericWebApplicationContext   : Exception encountered during context initialization ...
2020-03-26 17:56:27.380 ERROR 15845 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.beans.factory.UnsatisfiedDependencyException:
                Error creating bean with name 'org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration':
                                Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException:
                                                Error creating bean with name 'dataSource' defined in class path resource [org/springframework/boot/autoconfigure/jdbc/DataSourceConfiguration$Hikari.class]:
                                                                Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException:
                                                                                Error creating bean with name 'org.springframework.boot.autoconfigure.jdbc.DataSourceInitializerInvoker': 
                                                                                                Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: 
                                                                                                                Could not initialize class sun.security.provider.NativePRNG

In my pom.xml file I have three dependencies for these drivers (sybase, mssql and h2). If I remove the sybase OR the mssql dependency (thus only have one driver + the H2 database driver) the database setup is successful and all unit test run/pass in Jenkins.

This class that is supposedly not found does exists in the Java 11 JDK's that we use (locally it is one for windows, on Jenkins it is the same jdk but for linux).

Some questions I now have:

• I'm completely lost as to why the datasource requires this class, and only on linux machines.
• This class is an empty shell on windows, but it has an implementation on linux (api here). What does this class try to do?
• Why can't this class load it properly.
• I also don't get why this problem only occurs when multiple drivers are on the classpath, but it doesn't occur when only one is present.

I found only one blogpost about this topic making the undocumented claim that Spring doesn't support multiple drivers:

One of the problems with Spring is, that you can’t actually use multiple database drivers in the same package. That’s just a limitation Spring has. But fortunately for us, someone, way smarter than me, figured out how you can do the next best thing: separate repositories with different database drivers.

I think this is false because the application runs fine. I don't quite get why building on Jenkins screws up the unit tests. The solution provided in this blogpost won't work for me because I only have 1 package with 1 logic, I don't need these multiple drivers to access multiple datasources.

Answer 1

Your problem is unusual and not easily reproducible so I have not a proven solution but some important hint.

First of all a common pitfall: NoClassDefFoundError does not happen only when a class is not in the classpath but also in case of an unchecked exception thrown by the static initializer of the class. This fact is not reported in the Javadoc but can be easily demonstrated with this code:

public class Demo {

    public static void callMe() {
        System.out.println("Called");
    }

    static {
        if (true) {
            System.err.println("Going to fail");
            throw new NullPointerException();
        }
    }
}


public class Main {

    public static void main(String[] args) {
        try {
            Demo.callMe();
        } catch(Throwable e) {
            e.printStackTrace();  //Log ExceptionInInitializerError and continue
        }
        System.err.println("2nd call");
        Demo.callMe();  //NoClassDefFoundError here
    }

}

NativePRNG is almost empty in Windows but the Linux implementation has a lot of code executed at classloading time and this code accesses /dev/random and/or /dev/urandom so a lot of things can go wrong. Fortunately it has some logging that can be activated with

-Djava.security.debug="provider,engine=SecureRandom"

and this could help troubleshooting a lot.

Other option: this post explains that you can change the default secure random implementation. Maybe the so-called SHA1PRNG implementation works.

The error appears when you have both sybase and mssql driver in classpath, but during unit tests and you say that in tests you are using H2 only. Note that modern JDBC drivers (v4.0+) are autodiscovered and automatically loaded by the classloader (the old Class.forName() trick is no more necessary, see Java tutorial). Any JDBC driver has a static initializer that, at least, notifies the DriverManager of its existence but could also access NativePRNG.

Now some guessing. In my example above you get NoClassDefFoundError on the 2nd access to the class. I guess that the first driver that tries to use that class gets ExceptionInInitializerError and silently ignores it (maybe it fallbacks to Math.random()), then the 2nd (or the 3rd) driver tries to do the same but gets NoClassDefFoundError and fails. This would be a bug in the driver (it should ignore NoClassDefFoundError too). So, if any other option fails, you could try to use a different version (newer or even older) of those drivers or also a different version of Hikari or a different DB pool.