Deserializing Protobuf-Net objects using a filter

Question

I am using protobuf-net in one of our projects to serialize/deserialize a large set of homogeneous objects. It is working quite well and the speed is fantastic. Just have one question though. While deserializing is it possible to use linq (or any other mechanism) to specify a filter criteria, so that objects which only meet that criteria are loaded? It is fairly trivial to deserialize ALL the objects and then apply a linq filter but I want to reduce the number of objects loaded into memory. The filter criteria can be fairly dynamic, so a string kind of mechanism would be fantastic (something like dlinq?).

Answer 1

No there is nothing built in, but if you have a clearly defined use-case it is something I could of course look at (I'm the author).

For now, I would suggest using some variant of:

var found = Serializer.DeserializeItems<A>(source, PrefixStyle.Base128,
         Serializer.ListItemTag).FirstOrDefault(obj => obj.Foo = "bar");

if(found != null) {...}

which will short-circuit when a match is found, and will release the objects for collection promptly (hopefully in gen-0). Or for multiple items, perhaps:

var list = Serializer.DeserializeItems<A>(source, PrefixStyle.Base128,
         Serializer.ListItemTag).Where(obj => obj.Foo = "bar").ToList();

(which again releases the non-matching items promptly)

To do this in the general case (especially for more complex queries) I can't think of a sane way to do it without materializing the object, so this is probably as close as you can get unless there is a very specific (and simple) scenario that happens to align nicely with the underlying data storage (for example, the filter is always on "tag 1").

Answer 2

Generally, you won't be able to treat your serialised data as anything but raw data until you've finished deserialising. You may get some memory benefits from filtering as you go, but unless this is a problem it wouldn't be worth bothering.

You may be able to filter the incoming serialised representation, but weigh the amount of time and effort to do this against what you will save.

Most of the time (in a desktop or server environment) it's best to go the simple option that works, then go to something more complex if you need it later.