Node JS upload file streams over HTTP

Question

I'm switching one of my projects from request over to something a bit more light-weight (such as got, axios, or fetch). Everything is going smoothly, however, I'm having an issue when attempting to upload a file stream (PUT and POST). It works fine with the request package, but any of the other three return a 500 from the server.

I know that a 500 generally means an issue on the server's end, but it is consistent only with the HTTP packages that I'm testing out. When I revert my code to use request, it works fine.

Here is my current Request code:

Request.put(`http://endpoint.com`, {
  headers: {
    Authorization: `Bearer ${account.token.access_token}`
  },
  formData: {
    content: fs.createReadStream(localPath)
  }
}, (err, response, body) => {
  if (err) {
    return callback(err);
  }

  return callback(null, body);
});

And here is one of the attempts using another package (in this case, got):

got.put(`http://endpoint.com`, {
  headers: {
    'Content-Type': 'multipart/form-data',
    Authorization: `Bearer ${account.token.access_token}`,
  },
  body: {
    content: fs.createReadStream(localPath)
  }
})
  .then(response => {
    return callback(null, response.body);
  })
  .catch(err => {
    return callback(err);
  });

Per the got documentation, I've also tried using the form-data package in conjunction with it according to its example and I still get the same issue.

The only difference between these 2 I can gather is with got I do have to manually specify the Content-Type header otherwise the endpoint does give me a proper error on that. Otherwise, I'm not sure how the 2 packages are constructing the body with the stream, but as I said, fetch and axios are also producing the exact same error as got.

If you want any of the snippets using fetch or axios I'd be happy to post them as well.

Answer 1

I know this question was asked a while ago, but I too am missing the simple pipe support from the request package

const request = require('request');

request
  .get("https://res.cloudinary.com/demo/image/upload/sample.jpg")
  .pipe(request.post("http://127.0.0.1:8000/api/upload/stream"))


// Or any readable stream
fs.createReadStream('/Users/file/path/localFile.jpeg')
  .pipe(request.post("http://127.0.0.1:8000/api/upload/stream"))

and had to do some experimenting to find similar features from current libraries.

Unfortunately, I haven't worked with "got" but I hope the following 2 examples help someone else that are interested in working with the Native http/https libraries or the popular axios library

---

HTTP/HTTPS

Supports piping!

const http = require('http');
const https = require('https');

console.log("[i] Test pass-through: http/https");

// Note: http/https must match URL protocol
https.get(
  "https://res.cloudinary.com/demo/image/upload/sample.jpg",
  (imageStream) => {
    console.log("    [i] Received stream");

    imageStream.pipe(
      http.request("http://localhost:8000/api/upload/stream/", {
        method: "POST",
        headers: {
          "Content-Type": imageStream.headers["content-type"],
        },
      })
    );
  }
);

// Or any readable stream
fs.createReadStream('/Users/file/path/localFile.jpeg')
  .pipe(
    http.request("http://localhost:8000/api/upload/stream/", {
      method: "POST",
      headers: {
        "Content-Type": imageStream.headers["content-type"],
      },
    })
  )

Axios

Note the usage of imageStream.data and that it's being attached to data in the Axios config.

const axios = require('axios');

(async function selfInvokingFunction() {
  console.log("[i] Test pass-through: axios");

  const imageStream = await axios.get(
    "https://res.cloudinary.com/demo/image/upload/sample.jpg",
    {
      responseType: "stream", // Important to ensure axios provides stream
    }
  );

  console.log("  [i] Received stream");

  const upload = await axios({
    method: "post",
    url: "http://127.0.0.1:8000/api/upload/stream/",
    data: imageStream.data,
    headers: {
      "Content-Type": imageStream.headers["content-type"],
    },
  });

  console.log("Upload response", upload.data);
})();