Send huge data from client to server

Question

I am working with Go and Grpc. I have one XML file to be sent from client to server. When server calls a method on client, client responds back with an XML document which could range from few kB to 100s MB or may be more. Please help me with how should I do it? Should I

1. assign the value to a string variable and send normally -> rpc GetCMData (CmRequest) returns (CmReply) {}
2. assign the value to a string variable and send as a stream -> rpc GetCMData (CmRequest) returns (stream CmReply) {}
3. Is there any other better way to do so?

This is how CmReply looks like:

message CmReply {
  string name = 1;
}

Answer 1

The second one is more the way you want to go. But another point: Are you sure you want to have these hundreds of megabytes in your RAM? I'm not the Go-Geek, but I generally would prefer to work on a File Stream. And probably implement something like a back-pressure-system.

If you work with streams: I would also implement something to make sure that you can handle a break of the stream. Let's say you have the half of your stream and the stream breaks. You don't want to start again and get all previous stream elements, which you already have on the client side. You risk again a connection break within the stream. Maybe at the same point. So, something like pagination. Let's say

message CMRequest{
  int startAt = 0;
}

message CmReply {
  string id = 0;
  string name = 1;
}

If the stream breaks, you send a new request with startAt= LastCmReply.id+1

Here's a video of some good practices in grpc: https://www.youtube.com/watch?v=Z_yD7YPL2oE&index=17&list=WL