Why to use marshal to dump and pickle to load?

Question

I'm trying to understand the python2 library remotely which helps to run code remotely through xmlrpc.

On the client, the author dump the object with marshal and load the result sent back from the server with pickle:

def run(self, func, *args, **kwds):
        code_str = base64.b64encode(marshal.dumps(func.func_code))
        output = self.proxy.run(self.api_key, self.a_sync, code_str, *args, **kwds)
        return pickle.loads(base64.b64decode(output))

And on the server side, he's doing the other way around:

def run(self, api_key, a_sync, func_str, *args, **kwds):
        #... truncated code
        code = marshal.loads(base64.b64decode(func_str))
        func = types.FunctionType(code, globals(), "remote_func")
        #... truncated code
        output = func(*args, **kwds)
        output = base64.b64encode(pickle.dumps(output))
        return output

What's the purpose of dumping with marshal and loading the result with pickle? (and vice-versa)

Answer 1

The object that is getting sent first using marshal is of a very specific type. It's a code object, and only that type needs to be supported. That is a type that the marshal module is designed to handle. The return value on the other hand, can be of any type, it's determined by what the func function returns. The pickle module has a much more general protocol that can serialize many different types of objects, so there's a decent chance it will support the return value.

While you could use pickle for both data items being passed between your programs, the marshal module's output is a little more compact and efficient for passing code objects, as pickle just wraps around it. If you try dumping the same code object with both marshal and pickle (using protocol zero, the default in Python 2), you'll see the output from marshal inside the output from pickle!

So to summarize, the marshal module is used to sent the code because only code objects need to be sent and it's a lower-level serialization, which can be a little more efficient some of the time. The return value is sent back using pickle because the program can't predict what type of object it will be, and pickle can serialize more complicated values than marshal can, at the cost of some additional complexity and (sometimes) the size of the serialization.