JVM crashes while authenticating pub/sub

Question

I use GCP client libraries to implement pub/sub model in my spring-boot application. For authenticating i'm using GOOGLE_APPLICATION_CREDENTIALS path env variable. It works fine with other versions of JDK/JRE, But it fails with segmentation Error with below mentioned jdk/jre

Environment details

Java version:

openjdk version "1.8.0_322"
OpenJDK Runtime Environment (Zulu 8.60.0.22-SA-linux-musl-x64) (build 1.8.0_322-b06)
OpenJDK 64-Bit Server VM (Zulu 8.60.0.22-SA-linux-musl-x64) (build 25.322-b06, mixed mode)

Log:

# A fatal error has been detected by the Java Runtime Environment: 
# SIGSEGV (0xb) at pc=0x0000000000003fd6, pid=1, tid=0x00007f99a14fcb38 
# 
# JRE version: OpenJDK Runtime Environment (Zulu 8.60.0.22-SA-linux-musl-x64) (8.0_322-b06) (build 1.8.0_322-b06) 
# 
# Java VM: OpenJDK 64-Bit Server VM (25.322-b06 mixed mode linux-amd64 compressed oops) 
# Problematic frame: 
# C 0x0000000000003fd6 
# 
# Core dump written. Default location: //core or core.1 
# 
# An error report file with more information is saved as: 
# /tmp/hs_err_pid1.log 
# 
# If you would like to submit a bug report, please visit: 
# http://www.azul.com/support/


Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  java.lang.ClassLoader$NativeLibrary.load(Ljava/lang/String;Z)V+0
j  java.lang.ClassLoader.loadLibrary0(Ljava/lang/Class;Ljava/io/File;)Z+328
j  java.lang.ClassLoader.loadLibrary(Ljava/lang/Class;Ljava/lang/String;Z)V+92
j  java.lang.Runtime.load0(Ljava/lang/Class;Ljava/lang/String;)V+57
j  java.lang.System.load(Ljava/lang/String;)V+7
j  io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(Ljava/lang/String;Z)V+5
v  ~StubRoutines::call_stub
J 2066  sun.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (0 bytes) @ 0x00007f5cad99bdf7 [0x00007f5cad99bd80+0x77]
J 2065 C1 sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (104 bytes) @ 0x00007f5cad9a2a8c [0x00007f5cad9a1900+0x118c]
J 1974 C1 sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (10 bytes) @ 0x00007f5cad961784 [0x00007f5cad961680+0x104]
J 2084 C1 java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (62 bytes) @ 0x00007f5cad9a3e8c [0x00007f5cad9a3aa0+0x3ec]
j  io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader$1.run()Ljava/lang/Object;+53
v  ~StubRoutines::call_stub
J 1349  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object; (0 bytes) @ 0x00007f5cad764f4f [0x00007f5cad764f00+0x4f]
j  io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(Ljava/lang/Class;Ljava/lang/String;Z)V+10
j  io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(Ljava/lang/ClassLoader;Ljava/lang/String;Z)V+15
j  io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(Ljava/lang/String;Ljava/lang/ClassLoader;)V+359
j  io.grpc.netty.shaded.io.netty.channel.epoll.Native.loadNativeLibrary()V+60
j  io.grpc.netty.shaded.io.netty.channel.epoll.Native.<clinit>()V+76
v  ~StubRoutines::call_stub
j  io.grpc.netty.shaded.io.netty.channel.epoll.Epoll.<clinit>()V+28
v  ~StubRoutines::call_stub
J 993  java.lang.Class.forName0(Ljava/lang/String;ZLjava/lang/ClassLoader;Ljava/lang/Class;)Ljava/lang/Class; (0 bytes) @ 0x00007f5cad6995fa [0x00007f5cad699580+0x7a]
J 1952 C1 java.lang.Class.forName(Ljava/lang/String;)Ljava/lang/Class; (15 bytes) @ 0x00007f5cad948d4c [0x00007f5cad948ba0+0x1ac]
j  io.grpc.netty.shaded.io.grpc.netty.Utils.isEpollAvailable()Z+3
j  io.grpc.netty.shaded.io.grpc.netty.Utils.<clinit>()V+144
v  ~StubRoutines::call_stub
j  io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder.<clinit>()V+16
v  ~StubRoutines::call_stub
j  io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForAddress(Ljava/lang/String;I)Lio/grpc/netty/shaded/io/grpc/netty/NettyChannelBuilder;+2
j  io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForAddress(Ljava/lang/String;I)Lio/grpc/ManagedChannelBuilder;+3
j  io.grpc.ManagedChannelBuilder.forAddress(Ljava/lang/String;I)Lio/grpc/ManagedChannelBuilder;+5
j  com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel()Lio/grpc/ManagedChannel;+285
j  com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$$Lambda$596.createSingleChannel()Lio/grpc/ManagedChannel;+4
j  com.google.api.gax.grpc.ChannelPool.<init>(Lcom/google/api/gax/grpc/ChannelPoolSettings;Lcom/google/api/gax/grpc/ChannelFactory;Ljava/util/concurrent/ScheduledExecutorService;)V+71
j  com.google.api.gax.grpc.ChannelPool.create(Lcom/google/api/gax/grpc/ChannelPoolSettings;Lcom/google/api/gax/grpc/ChannelFactory;)Lcom/google/api/gax/grpc/ChannelPool;+9
j  com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel()Lcom/google/api/gax/rpc/TransportChannel;+10
j  com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel()Lcom/google/api/gax/rpc/TransportChannel;+35
j  com.google.api.gax.rpc.ClientContext.create(Lcom/google/api/gax/rpc/StubSettings;)Lcom/google/api/gax/rpc/ClientContext;+179
j  com.google.cloud.pubsub.v1.stub.GrpcSubscriberStub.create(Lcom/google/cloud/pubsub/v1/stub/SubscriberStubSettings;)Lcom/google/cloud/pubsub/v1/stub/GrpcSubscriberStub;+6
j  com.google.cloud.pubsub.v1.Subscriber.doStart()V+16
j  com.google.api.core.AbstractApiService$InnerService.doStart()V+4
j  com.google.common.util.concurrent.AbstractService.startAsync()Lcom/google/common/util/concurrent/Service;+33
j  com.google.api.core.AbstractApiService.startAsync()Lcom/google/api/core/ApiService;+4
j  com.google.cloud.pubsub.v1.Subscriber.startAsync()Lcom/google/api/core/ApiService;+1

Dependencies:

        <dependencies>
            <dependency>
                <groupId>com.google.cloud</groupId>
                <artifactId>libraries-bom</artifactId>
                <version>25.1.0</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-pubsub</artifactId>
        </dependency>
   <dependencies>

And also i wanted to know is there any other way to authenticate other than using path env variable? Can I use spring.cloud.gcp.credentials.location=file:{location} with GCP client libraries, instead of env variable?

Answer 1

As mentioned by @Juraj Martinka, it was problem with underlying google library io.grpc.netty.shaded. It seems Netty does not support Alpine since Netty depends on glibc but Alpine does not have it, it has musl libc instead.

The issue disappears if you disable Netty's native support or if you use an image that has glibc, e.g.:

azul/zulu-openjdk-alpine:11-jre: Alpine-based, no glibc -> does not work
azul/zulu-openjdk:11: Ubuntu-based, has glibc -> works

Using -Dio.grpc.netty.shaded.io.netty.transport.noNative=true avoids the segfault
example:

java -jar -Dio.grpc.netty.shaded.io.netty.transport.noNative=true app.jar

The other workaround is, using grpc-netty instead of grpc-netty-shaded

<dependency>
  <groupId>com.google.cloud</groupId>
     <artifactId>google-cloud-pubsub</artifactId>
  <exclusions>
    <exclusion>
      <groupId>io.grpc</groupId>
      <artifactId>grpc-netty-shaded</artifactId>
    </exclusion>
  </exclusions>
</dependency>

<dependency>
  <groupId>io.grpc</groupId>
  <artifactId>grpc-netty</artifactId>
</dependency>

Reference Links: Link 1, Link 2