How to gzip and post a file without memory allocation

Question

Does anyone know how I can open a file, gzip it on the go and send it to a server using a post request?

I have a file that is a reader. Also, I have http.Post() function where I can pass the reader as a 3rd parameter. But gzip.NewWriter() expects a writer and returns a writer.

I know that I can create some intermediary buffer to connect this all but I don't want to allocate much memory because these files can be quite big. Is there a way to make this pipe and pass it to the HTTP post request?

Answer 1

Other answers outline how to use io.Pipe. This answer shows more detail, particularly with regards to error handling.

func gzFileReader(fname string) (io.ReadCloser, error) {
    f, err := os.Open(fname)
    if err != nil {
        return nil, err
    }

    // Use io.Pipe and a goroutine to create reader
    // on data written by the appliation.
    r, w := io.Pipe()
    go func() {
        // Always close the file.
        defer f.Close()

        // Copy file through gzip to pipe writer.
        gzw := gzip.NewWriter(w)
        _, err := io.Copy(gzw, f)

        // Use CloseWithError to propgate errors back to
        // the main goroutine.
        if err != nil {
            w.CloseWithError(err)
            return
        }

        // Flush the gzip writer.
        w.CloseWithError(gzw.Close())
    }()
    return r, nil
}

Use the function like this:

body, err := gzFileReader("example.txt")
if err != nil {
    log.Fatal(err)
}
defer body.Close()

req, err := http.NewRequest("POST", "http://example.com/", body)
if err != nil {
    log.Fatal(err)
}
req.Header.Set("Content-Encoding", "gzip")
resp, err := http.DefaultClient.Do(req)

Answer 2

Yes, io.Pipe

    reader, writer := io.Pipe()
    go func() {
        defer w.Close()

        gw := gzip.NewWriter(w)
        defer gw.Close()

        gw.Write( []byte("replace with your data"))
    }()

    http.Post("some.url", "application/zip", reader)