Generating python client from a .proto with (well-known) timestamp

Question

When I take a simple hello.proto file

syntax = "proto3";
package helloworld;
//import "timestamp.proto";
service Greeter {
  rpc SayHello (HelloRequest) returns (HelloReply) {}
}
message HelloRequest {
  string name = 1;
  //google.protobuf.Timestamp smth = 2;
}
message HelloReply {
  string message = 1;
}

and (within virtualenv -p python3 py) generate python code with

python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. ./hello.proto

I get two nice files: hello_pb2.py and hello_pb2_grpc.py

and a simple two-line playground.py:

import hello_pb2
import hello_pb2_grpc

works fine with python playground.py

Everything changes when try to use a well-known protobuf type, timestamp, and uncomment lines 3 and 9.

---

The closest corresponding file, timestamp.proto I see in site-packages:

$ find -name timestamp\*
./lib/python3.5/site-packages/grpc_tools/_proto/google/protobuf/timestamp.proto
./lib/python3.5/site-packages/google/protobuf/timestamp_pb2.py
./lib/python3.5/site-packages/google/protobuf/__pycache__/timestamp_pb2.cpython-35.pyc

so I use a bit different code generation command:

python -m grpc_tools.protoc -I. \
       -I./lib/python3.5/site-packages/grpc_tools/_proto/google/protobuf \
       --python_out=. --grpc_python_out=. ./hello.proto

but python playground.py now results in an error:

Traceback (most recent call last):
  File "playground.py", line 1, in <module>
    import hello_pb2
  File "/py/hello_pb2.py", line 16, in <module>
    import timestamp_pb2 as timestamp__pb2
ImportError: No module named 'timestamp_pb2

that's because hello_pb2.py now has a line:

import timestamp_pb2 as timestamp__pb2

I tried a bit different playground.py:

import sys
sys.path.append('./lib/python3.5/site-packages/google/protobuf')
import hello_pb2
import hello_pb2_grpc

but that leads to a different error:

$ python playground.py 
Traceback (most recent call last):
  File "playground.py", line 3, in <module>
    import hello_pb2
  File "/py/hello_pb2.py", line 25, in <module>
    dependencies=[timestamp__pb2.DESCRIPTOR,])
  File "/py/lib/python3.5/site-packages/google/protobuf/descriptor.py", line 829, in __new__
    return _message.default_pool.AddSerializedFile(serialized_pb)
TypeError: Couldn't build proto file into descriptor pool!
Invalid proto descriptor for file "hello.proto":
  hello.proto: Import "timestamp.proto" has not been loaded.
  helloworld.HelloRequest.smth: "google.protobuf.Timestamp" seems to be defined in "google/protobuf/timestamp.proto", which is not imported by "hello.proto".  To use it here, please add the necessary import.

So, I'm out of ideas. How do I use this well-known type with python (3)?

---

Details: ubuntu:xenial, python 3.5.2, virtualenv 15.2.0, env: virtualenv -p python3 py && cd py && source bin/activate && pip install grpcio grpcio_tools (this installs protobuf-3.5.2, grpcio-1.10.0, grpcio_tools-1.10.0)

Answer 1

Try import "google/protobuf/timestamp.proto";

This shouldn't happen using the Python grpc_tools, but for other distributions of protoc, protoc might complain, that it does not know where "google/protobuf/timestamp.proto" is. In such case you will need to specify the root to the built-in protobuf files using an -I flag.

Why does this work?

Based on my experience with protoc (aka. not confirmed with documentation or source code), it looks like when .proto files are compiled to Python, the import statements in .proto change from:

import "path/to/file.proto";

To:

import path.to.file_pb2

The protobuf compiler does not try to resolve the location of the files (which makes sense, since it is unaware of your Python environment, and does not know where other compiled .proto files are).

As of 4/9/2018 the pip grpc package includes built-in protobuf types from the google.protobuf package. That is why we want our imports to include the google/protobuf prefix.

Side note

Protobuf's import resolution can be especially annoying if you want the root of your compiled .proto project to be different than the root of your Python project (e.g. you might want to store all your protos in a protos package). So far, the only solutions I found to that were:

• Hack around protoc and add the new root to all import statements (e.g. change import "path/to/file.proto" to import "protos/path/to/file.proto"). This can be done programmatically before each compilation.

• Add the root of your compiled proto project to your Python path:

  import sys
  sys.path.append('path/to/protos_root/')