Mixing curls --data-binary and the ability to encode url-search params

Question

I want to make an post request using curl. The body is a binary file but I also want curl to encode a url-search parameter.

curl --request POST \
     --header 'Content-Type:application/octet-stream' \
     --data-binary "@file.txt" \ # The body part, raw, not encoded
     --data-urlencode "name=john" \ # The url-search-param part, encoded
     "https://example.com"

The problem is that curl treats --data-urlencode as part of the body, what I want is that name=john gets appended to the url part like this: https://example.com?name=john whereby the --data-binary part gets send over as the POST body. (in reality the search-parameter is a string with "invalid" url-characters which need to be encoded)

TLDR: I want to use --data-urlencode as if I'm making a GET request (to append the parameter) and the POST --data-binary to set the actual POST body.

A quick google search gives me the info to use --get / -G but this transforms the request into a GET request what I don't want so there are already a dozen questions about this on SO (but none of them cover my case):

• Is it possible to use `--data-urlencode` and `--data-binary` options for the same curl command? - Different issue, he wants the --data-binary thing to be part of the url (?!).
• How to urlencode data for curl command? - Yeah I want that, but also sending a binary file using POST...
• CURL Command Line URL Parameters Well yes, but I want a file as body and It's a POST request in my case...
• How to urlencode url params with a curl POST but not urlencode the body data? - Exactly what I need but again, curl will tread both --data-urlencode and --data-binary as part of the body if -X POST is used. A look at the comment section of the answer makes me thing that this isn't possible with curl?!

... and many other duplicates of how to just encode the body or url part.

Answer 1

You can't. (until curl 7.87.0 ships with the new --url-query option)

The --data* options are used to either build the request body, or you can ask curl to put them all as query parameters in the URL (with -G). Those options cannot build both, in a single command line.

My advice would be to build the URL "manually" and keep the --data options to build the request body, mostly because the URL part tends to be easier and smaller.

Until curl 7.87.0 is installed on your machine, because once it does you can use --url-query and --data-urlencode and happiness shall ensue.

Answer 2

curl won't mix post data and query strings for you.

I found a convenient solution for keeping everything in a single bash script without adding additional dependencies. If python, ruby, or node is available, they have url encoding libraries.

Must construct urlencoded query string yourself

A bash only solution for constructing the query string was provided by Chris Down. It is useful gist with an urlencode function that I found suitable. The comments of his gist have modifications for use in other shells.

# From Chris Down https://gist.github.com/cdown/1163649
urlencode() {
    # urlencode <string>
    old_lc_collate=$LC_COLLATE
    LC_COLLATE=C
    local length="${#1}"
    for (( i = 0; i < length; i++ )); do
        local c="${1:$i:1}"
        case $c in
            [a-zA-Z0-9.~_-]) printf '%s' "$c" ;;
            *) printf '%%%02X' "'$c" ;;
        esac
    done
    LC_COLLATE=$old_lc_collate
}

# Construct query string parameter
VAL_1="http://localhost:3030/ds/spek"
ENC_1=$(urlencode "${VAL_1}")
PARAM_1="graph=${ENC_1}"

curl -X PUT --data-binary "@${SPEK_FILE}" \
  --header 'Content-type: application/ld+json' \
  "http://localhost:3030/ds?${PARAM_1}"

# > Content-type: application/ld+json
# > Content-Length: 1215

Using both options appends the data to the body.

Attempting to use both --databinary and --data-encode in a POST ends up with both being appended to the body. Notice the content length difference with the above example.

curl -vvvv -X PUT --data-binary "@${SPEK_FILE}" \
  --header 'Content-type: application/ld+json' \
  --data-urlencode "graph=http://localhost:3030/ds/spek" \
  'http://localhost:3030/ds'

# > Content-type: application/ld+json
# > Content-Length: 1263