Partial input

(Helge Heß) #21

I've looked at the encoding and unless I've missed something that doesn't require any framing at all (which would have been very surprising for such a format in the first place).
So I assume that this is just an API issue and you won't support reporting remaining Data?

(Tim Kientzle) #22

If you've read the encoding description, then you should be able to understand the following example:

08 01 10 02

This could encode one message, two messages, or even three messages. (In protobuf, a valid encoded message can have no bytes at all.) There is no way to know which case applies without additional information.

(Helge Heß) #23

I honestly can't follow you. We are talking about SwiftProtobuf here, aren't we? So if I do a say Movie(serializedData: receivedData), in which case would the result of that depend on the size of the received data?

Do we agree that this is: in no case?

As far as I can see there are only three states:

  1. insufficient data to decode Movie
  2. exact amount of data to decode Movie
  3. more data than Movie ( Movie, RemainingData )

We need API for 3. here to avoid unnecessary framing.

Presumably your "additional information" is the type, isn't it? That is always available to SwiftProtobuf by the means of the schema, no? Even if not, how would the content-length help you in any way?
As far as I can see there is no not-length-delimited item in the wire protocol.

Just asking, maybe I'm really missing something.

(Tim Kientzle) #24

Protobuf does frame each field but does not frame a message. In the short example I gave, 08 01 encodes one field and 10 02 encodes another field. But there is no way to tell whether both fields are for the same message or not.

If you are using protobuf, you are responsible for framing messages because protobuf does not do it for you.

(Helge Heß) #25

But the type frames a message, doesn't it?

(Helge Heß) #26

It does. So can we get an API in which the decoder returns the message and the bytes it didn't consume, or is this something you won't do?

(Tim Kientzle) #27

No, it doesn't.

(Helge Heß) #28

What is so hard about: "No, it doesn't", because:

 ABC.

:slight_smile:

Never mind.

Summary: For SwiftProtobuf one needs framing. API or other reasons.

(Itai Ferber) #29

I can't claim to be familiar with exact Protobuf terminology, but as an outsider, if I've understood you correctly, it seems like you're asserting that for your use-case, every message contains a single top-level type (e.g. Movie), correct? Such that what you're looking to do is to decode a Movie from a stream, and if there's any leftover data, use it to decode the next Movie?

If I've understood correctly, then the mismatch here is that 1 top-level type == 1 message is not necessarily true in general in protobuf, and if you've got a message format which sends several top-level types sequentially, you've no way of knowing when a given message ends short of framing it. In your case, you are trying to avoid this framing because of this correspondence.

Am I off base with the phrasing here, @tbkka?

(Helge Heß) #30

it seems like you're asserting that for your use-case, every message contains a single top-level type (e.g. Movie), correct?

Well, I'd say that is how Protobuf works. When decoding an item, you give it one specific top level type to decode. That could be an enum, but it is always a discrete type, right?

Such that what you're looking to do is to decode a Movie from a stream, and if there's any leftover data, use it to decode the next Movie?

That is the idea indeed.

If I've understood correctly, then the mismatch here is that 1 top-level type == 1 message is not necessarily true in general in protobuf

Why not? My understanding is that this is the case. Protobuf doesn't encode any type data but just raw struct data. If the client encodes a Movie, the server needs to decode a Movie.

and if you've got a message format which sends several top-level types sequentially, you've no way of knowing when a given message ends short of framing it.

I don't understand this part. If I would have to dispatch on types, framing is not enough, the client would also need to transport the type of the message in the frame.
But this is not how SwiftProtobuf works. In SwiftProtobuf you e.g. do:

Movie(decoding: data)

thereby fixing the type and specifying the frame size.

So far I've heard nothing which disputes that. (well, @tbkka says that, but doesn't outline why: "But the type frames a message, doesn't it?" => "No, it doesn't." I don't care too much but I don't understand why that is)

(Tim Kientzle) #31

The "why": The people who designed protobuf assumed that there would be some other mechanism tracking the start and end of each message. Because of that assumption, they were able to make a lot of simplifications that led to a very compact and high-performance serialization system. But this also means that protobuf does require that you have some other mechanism to determine message boundaries.

To fully understand it, you need to study the encoding and look at examples. But I'll give it one more try:

Binary protobuf does not really encode a "message"; it just encodes "fields." Protobuf assumes that "messages" are framed by some other mechanism.

A protobuf encoder works by writing out a bunch of fields. There is no message start marker, no message end marker, nor any other extraneous overhead. The encoder does not have to write every field; it may not write any of them. The new proto3 variant, for example, does not write out a field if the value is the default. This can save a lot of space. The encoder is allowed to add extra fields or write the fields in different orders. Allowing encoders to add extra fields helps support schema evolution; some systems may know about new fields that old systems don't know about. Allowing fields to be out-of-order makes it easier for systems to decode, store, and re-encode fields they don't understand.

A protobuf decoder is told what type to expect: It then proceeds to decode fields until it runs out of data or it finds something that is malformed. There is literally nothing in the data itself that marks the beginning or end of a message. As mentioned above, fields can be omitted or duplicated and can appear in any order.

If you give a protobuf decoder a block of data that includes a complete message plus extra data at the end, the decoder will decode the extra data as additional fields for the same message. It has no reason to stop decoding unless the data is malformed. If you concatenate two messages into a buffer and try to decode them, the decoder will simply decode the combined fields as if it were a single message. This is well-documented and surprisingly useful behavior.

(Helge Heß) #32

How is that useful? And what happens with the extra data in the Movie example?

(Helge Heß) #33

But there is, it is inherent in the structure of the type being decoded?

Nothing in the wire protocol has a non-deterministic type, i.e. an open length. I.e. the binary size of a Movie or whatever is always going to be the same, it can never be longer or shorter. It always keeps being a Movie ;-) When you decode it from a Data, it can be too little, to much, or exactly right.
The question still is and was from the beginning, what is the API to reclaim the Data which was not being used when decoding the top-level type.

It is perfectly OK to say that you don't support that.

(Tim Kientzle) #34

Suppose I have a protobuf message with two fields:

message Foo {
   int32 a = 1;
   int32 b = 2;
}

Now I create two objects Foo(a=1) and Foo(b=2). The first one encodes as 08 01 (field #1 with value 1). The second encodes as 10 02 (field #2 with value 2). Note that the fields that don't have a value didn't get encoded.

If I send both of them over a network connection, the receiver may read 08 01 10 02. If you ask the decoder to decode Foo with that data, it will decode both fields into a single Foo object and you will end up with Foo(a=1, b=2). The decoder will not stop after 08 01; it has no way to know that the second field came from a different object.

3 Likes
(Helge Heß) #35

That makes sense, thanks for the explanation!

(Thomas Van Lenten) #36

One other reason protobuf doesn't frame messages, is it has a design goal to support easy addition to messages.

You can make one thing write out a message with some of the fields, and another process can write out other fields for the same message, and simply concatenate the two "blobs" and still have a valid, single message.

This can come in very handy with servers assembling things from multiple places. Likewise, it can let a server proxy a message adding in an extra context field(s) as needed. If there was framing on the message, it would require unwrapping the framing and rewrapping it.

(Helge Heß) #37

That also makes sense. (I didn't suggest that Protobuf does framing, I just assumed that the size of a message is implicit by the type)

(Tony Allevato) #38

Some language implementations of protobufs do provide an API for handling message streams a little more cleanly; for example, Java has a writeDelimitedTo method (and corresponding parse methods) that write a varint containing the length of the message, and then the varint itself, to a stream. Then you can just run a loop that writes those to the stream, and when you go back to read the stream, you can loop over it safely because the methods know when each message actually ends.

Those methods aren't an official part of the protobuf spec, though, which is why they weren't implemented right away. Feel free to file a feature request, though! If this is a use case that folks doing server work are having to invent for themselves, we may want to unify the implementation.

(Helge Heß) #39

Yes, that matches what Johannes has been showing above. Would be cool to have a recommendation on how to do this as part of protobuf itself (or a separate RFC, framing for protobuf).

Is there an informal "standard" on how to use HTTP w/ protobuf? Do you
a) use a single message per HTTP message
b) use multiple messages per HTTP message and do the int64-prefix?
c) use multipart in HTTP message to encapsulate multiple protobuf messages
d) do not combine HTTP w/ protobuf because that is just stupid for e.g. performance reasons

Or maybe using WebSockets?

(Thomas Van Lenten) #40

So the proto grammar has a service term, but is really is only useful to generate a pure rpc style interface. Historically they've kept out of transmission details as there are lots of options. Along the lines of the lack of "framing", a message (binary or JSON encoding), doesn't actually capture the message "type" either. Deferring to the transmission method to decide if they need this and how to deal with it.

Some of the more common methods I've seen:

  • HTTP "REST" api, but with a binary payload payload in the body of the POST/PUT and then binary in the reply.
  • gRPC with HTTP 2, side channel, etc.
  • Raw TCP sockets with varint for framing of each message.