Building an Android executable gRPC server that uses protocol buffers (without APK)

Question

I compiled the gRPC Android example from here.

I want to run the program as executable from adb shell.

Added these lines to grpc-helloworld.cc:

#include <iostream>

int main() {
  std::cout << "qwerty" << std::endl;
  return 0;
}

And these lines to its CMakeLists.txt:

add_executable(avocado
    src/main/cpp/grpc-helloworld.cc)

target_include_directories(avocado
  PRIVATE ${HELLOWORLD_PROTO_HEADERS})

target_link_libraries(avocado
  helloworld_proto_lib
  android
  ${log-lib})

Then I pushed the generated executable and libs file and tried to run it:

LD_LIBRARY_PATH=. ./avocado

I got the following error:

[libprotobuf FATAL /home/buga/grpc/third_party/protobuf/src/google/protobuf/stubs/common.cc:79] This program was compiled against version 3.0.0 of the Protocol Buffer runtime library, which is not compatible with the installed version (3.5.1). Contact the program author for an update. If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library. (Version verification failed in "out/soong/.intermediates/frameworks/av/drm/libmediadrm/libmediadrm/android_arm64_armv8-a_kryo300_shared_core/gen/proto/frameworks/av/drm/libmediadrm/protos/plugin_metrics.pb.cc".)terminating with uncaught exception of type google::protobuf::FatalException: This program was compiled against version 3.0.0 of the Protocol Buffer runtime library, which is not compatible with the installed version (3.5.1). Contact the program author for an update. If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library. (Version verification failed in "out/soong/.intermediates/frameworks/av/drm/libmediadrm/libmediadrm/android_arm64_armv8-a_kryo300_shared_core/gen/proto/frameworks/av/drm/libmediadrm/protos/plugin_metrics.pb.cc".) Aborted

What am I doing wrong?

We realized that there is a version of the protobuf library called libprotobuf-cpp-full.so and libprotobuf-cpp-lite.so, and it seems that their version is 3.0.0. This is conflicting with our version (3.5.1) which is compiled into either a static lib, or as a shared lib.

Answer 1

I'm not quite sure why this happens. Something about once the linker loads helloworld_proto_lib, it overrides all loaded protobuf symbols, and for some reason another library that you had nothing to do with crashes your program. But that's not telling you anything new.

Here's one way to solve this problem:

1. Changes to grpc-helloworld.cc

Make the main extern "C", and change its name maybe. For example:

 extern "C" int my_main() {
  std::cout << "qwerty" << std::endl;
  return 0;
}

2. Add file grpc-avocado.cc

This will contain the actual main of the executable, which will dynamically load the libraries helloworld_proto_lib and grpc-helloworld. Here's how to do it:

#include <iostream>
#include <android/dlext.h>
#include <dlfcn.h>

int main() {
  android_dlextinfo extinfo;
  extinfo.flags = ANDROID_DLEXT_FORCE_LOAD;

  void* proto_lib = android_dlopen_ext("/path/to/libhelloworld_proto_lib.so", RTLD_LAZY, &extinfo);
  void* helloworld = dlopen("/path/to/libgrpc-helloworld.so", RTLD_LAZY);
  int (*my_main)() = (int (*)())dlsym(helloworld, "my_main");

  return my_main();
}

The function android_dlopen_ext from #include <android/dlext.h>, and its flag argument, are described here: https://developer.android.com/ndk/reference/group/libdl . In the above code we pass the flag ANDROID_DLEXT_FORCE_LOAD, which is documented as:

When set, do not use stat(2) to check if the library has already been loaded.

This flag allows forced loading of the library in the case when for some reason multiple ELF files share the same filename (because the already-loaded library has been removed and overwritten, for example).

Note that if the library has the same DT_SONAME as an old one and some other library has the soname in its DT_NEEDED list, the first one will be used to resolve any dependencies.

I think the text in bold is what explains why this solution works.

3. Change CMakeLists.txt

Since you'll be loading helloworld_proto_lib dynamically, you can now remove it from the executable definition, and there's no need for any proto headers:

add_executable(avocado
    src/main/cpp/grpc-avocado.cc)

target_link_libraries(avocado
  android
  ${log-lib})

Build, push, and run

You can now build, push the executable avocado and the two libraries libgrpc-helloworld.so, libhelloworld_proto_lib.so, and run. You don't need LD_LIBRARY_PATH. Good luck with the rest of your project!