File already exists in database error from Protobuf when deploying Google Dataflow template after MacOS Catalina upgrade

Question

I have a project that uses Google Dataflow. I have been successfully using the following command (and commands like it) for months to deploy templates.

.venv/bin/python -m dataflow.registry_files.delimited_file --runner=DataflowRunner --region=us-central1 --project=myproject --staging_location=gs://mybucket-staging/staging/gr265 --template_location=gs://mybucket-code/templates/gr265 --temp_location=gs://mybucket-staging/temp/gr265 --no_use_public_ips --save_main_session --setup_file=dataflow/setup.py --projectId=myproject --datasetId=padl_staging --tableId=gr265 --configFile=gs://mybucket-code/registry/registry_files.yaml --fileType=gr265

This command continues to work on windows 10 and Debian machines in my team.

Since I upgraded to catalina (10.15.1, with python 3.7.5 and apache-beam==2.16.0 ) I get the following error:

[libprotobuf ERROR google/protobuf/descriptor_database.cc:58] File already exists in database: 
[libprotobuf FATAL google/protobuf/descriptor.cc:1370] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size): 
libc++abi.dylib: terminating with uncaught exception of type google::protobuf::FatalException: CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size): 
Abort trap: 6

I have done all of the following, with many reboots:

1. Run xcode-select --install

2. Run brew update-reset, brew update, brew upgrade, and brew reinstall python all to no effect (except after brew update-reset, brew doctor works again)

3. Run brew uninstall protobuf and brew install protobuf

4. Run pip3 uninstall protobuf outside of the virtual environments

5. Deleted and re-created my virtual environments from their requirements files.

6. Several bits of voodoo involving /usr/local/include that I located elsewhere on stack overflow that didn't help.

I wondered if this was just my machine, and unfortunately was able to reproduce it on the other macOS Catalina laptop in my team, but not the laptop still running macOS Mojave.

Answer 1

According with the Apache Beam Issue 8368 this issue is related with a pyarrow version, you have to test with pyarrow 0.15.1 beam since is the one which works on MacOS 10.15 as is mentioned in this link.

Please let us know how it works.