Protobuf NUL-framing

Question

Common practice for using protobufs over the wire, including by gRPC, is to length prefix protobuf messages into frames (e.g. like this) so that the decoder knows when one message stops and the next starts.

This seems unnecessary. According to the spec, a protobuf message is comprised of a sequence of tags followed by values:

message := (tag value)*
tag := (field << 3) bit-or wire_type

Once a tag is read, the length of the value is then known and the parser can parse the value in its entirety without needing more metadata. Thus, the length is only needed to figure out if at any given point there are more tags left to be parsed in the message.

An obvious non-length solution presents itself: null (0x0) termination. Field indices start at 1 so tag can never be 0; a flat INT encoding for field = 1 with a VARINT wire type produces 0b1000 = 8 and VARINT encoding will always set the MSB on the first byte and thus will always begin with a nonzero byte. Thus, if:

• the parser is in between tag-value pairs and
• encounters a null byte

it follows that this byte is not part of the rest of the protobuf message and thus a corresponding action (such as terminating the message) may be taken.

All of this seems very obvious given the protobuf specification, so am I simply missing some detail that breaks this?

Another way of phrasing the question is if there's a case where you will get a 0x0 tag within a valid message? It appears that nanopb would use NULL termination for a while and only stopped due to issues with debugging broken encoders.

Answer 1

The ultimate problem here is that it isn't defined as such in the specification, and any such change would break all existing implementations. The wire format doesn't allow for it, and encoding sub-messages in a null-suffixed way would require a new "wire type" so that decoders know to look for nulls in a forwards field-by-field way.

However! Something very similar already does exist; "groups". The wire specification defines groups as a prefix and suffix sentinel pair, akin to { and } in JSON. Not much more expensive than a null byte, although it needs two of them and they include the field number, but: virtually all decoders already know how to handle group encoding, even though they have been almost erased from sight.

To be clear: I'm not talking about the group concept from proto2 - I'm only talking about the wire specification. I would, for example, propose using instead a modifier in regular fields definitions, applicable only to message types.

I have tried to petition for their resurrection as an alternative mechanism for writing messages without needing to compute lengths, but: I have been unsuccessful so far. See: https://github.com/protocolbuffers/protobuf/issues/9134

Answer 2

I had to do this recently. It's a difficult problem. If you don't mind increasing the length of your data then you can turn each byte into two bytes thereby leaving a lot of possible values to serve as delimiters.

0xFF -> 0x0F, 0x0F

Now you'll find that data bytes can only occupy positions 0 to 15 and positions 16 to 255 are available for out-of-band use.

This does, of course, double the length of the data.

If you are writing your own protocol you only need to do this with variable data, constants that you control you could leave alone, though with protobuf you would have to do the whole message.

There's a lot of nice ASCII characters above 15, and field, record separators too. If you really need the delimiter to be 0x00 then you'll have to left shift all your nibbles to free up null:

0b1111_1111 -> 0b0001_1110, 0b0001_1110

and right shift it back before reassembly on receive, or just use the high nibble.

You might also experiment with a scheme that takes 3 bytes and encodes them as 4 leaving unused byte values to serve as delimiter, which will take more implementation but save bandwidth.