How to send a multipart streaming request using httpclient in ruby

Question

I'm trying to get a multipart streaming post working with HTTPClient for Ruby. Upon which I encounter two problems.

Problem 1: First, if I try to do a normal post via the way described in the docs; and I return it via httpbin.org I see this happening:

Code

        File.open(path_to_my_file) do |file|
          body = [{ 'Content-Type' => 'application/atom+xml; charset=UTF-8',
                    :content => '<entry>...</entry>' },
                  { 'Content-Type' => 'video/mp4',
                    'Content-Transfer-Encoding' => 'binary',
                    :content => file }]
          res = @http_client.post('http://httpbin.org/post', body: body)
          response = res
          puts res.body

Result

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "{\"Content-Type\"=>\"application/atom+xml; charset=UTF-8\", :content=>\"<entry>...</entry>\"}": "", 
    "{\"Content-Type\"=>\"video/mp4\", \"Content-Transfer-Encoding\"=>\"binary\", :content=>#<File:{{path_to_file}}>}": ""
  }, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "322", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Date": "Mon, 20 May 2019 06:43:17 GMT", 
    "Host": "httpbin.org", 
    "User-Agent": "HTTPClient/1.0 (2.8.3, ruby 2.6.0 (2018-12-25))"
  }, 
  "json": null, 
  "origin": "IP, IP", 
  "url": "https://httpbin.org/post"
}

As you can see, I cannot see the content of the file, but only the identifier containing the URI to the File; and as such, I don't know how to fix it so that I can see and return the content. As far as I know it looks like it tries to see the File object as a String, which is of course not what I'd like it doing.

Problem2: Whenever I create the body dynamically, that is; creating the hash-array dynamically with objects set in my code, and I try to send it asynchronously (which I believe is the correct way of getting it streaming in HTTPClient for Ruby, although I'm not entirely sure) it sends the entire body as data, and no longer as either a form or headers.

code

        request_body = []
        body.files.each do |k, v|
          request_body.push( { 'Content-Type' => v.content_type, :content => v.content })
        end
        body.values.each { |k, v| request_body << { k.to_sym => v }}

        #This creates the array correctly, although I just wanted to show how it was generated

        connection = @http_client.send(method + '_async', uri, body: request_body, header: headers)

        response = connection.pop
        # Starts reading result here

Response

  "args": {}, 
  "data": "%7B%22Content-Type%22%3D%3E%22video%2Fmp4%22%2C+%3Acontent%3D%3E%23%3CFile%3A%2Fhome%2Fuser%2Ffiles%2file.mp4%3E%7D=&%7B%3Avalue%3D%3E%22hello+world%22%7D=", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "247", 
    "Content-Type": "application/json", 
    "Date": "Mon, 20 May 2019 06:44:11 GMT", 
    "Host": "httpbin.org", 
    "User-Agent": "HTTPClient/1.0 (2.8.3, ruby 2.6.0 (2018-12-25))", 
    "ApplicationIdentifier": "identifier"
  }, 
  "json": null, 
  "origin": "IP, IP", 
  "url": "https://httpbin.org/post"
}

As you can see, it puts everything in data, and I honestly don't know how to get it to send the post with the body as a form instead of as data. I send the file here as a File object. I have tried sending it as a normal post instead of post_async, but it does not seem to be working.

Has anyone encountered these problems before and know how I should go about fixing these? (or can you see where I went wrong so that I can at least try getting a bit further)

Thanks in advance

Answer 1

Regarding your first issue, it appears that the documentation is not correct. I've performed some tests, and the content type appears to be application/x-www-form-urlencoded. You need to set the content type explicitly:

body: body, header: {'Content-Type': 'multipart/form-data; boundary=ABC'}

That's not sufficient though. You also need to set the content disposition manually. For instance:

      body = [{ 'Content-Type' => 'application/atom+xml; charset=UTF-8',
                'Content-Disposition' => 'form-data; name="name1"',
                :content => '<entry>...</entry>' },
              { 'Content-Type' => 'video/mp4',
                'Content-Disposition' => 'form-data; name="file"; filename="myfile.pdf"',
                'Content-Transfer-Encoding' => 'binary',
                :content => file }]

With this, httpbin reports one file and one form parameter.

Your second issue is related to the same. Both your files and values need to have the content disposition set. For instance (quick and dirty, style can probably be better):

files.each do |k, v|
    request_body << { 'Content-Type' => v.content_type, :content => v.content, 'Content-Disposition' => 'form-data; name="' + k + '"; filename="' + v.filename + '"' }
end
values.each { |k, v| request_body << { :content => v, 'Content-Disposition' => 'form-data; name="' + k + '"' }}

Note that you should escape any " in a name or filename.