Bash loop a curl request, output to file and stop until empty response

Question

So I have the following bash file and right now its looping a curl request based on the for loop. However, i am tying to find out how to continue looping until the response is empty.

Unfortunately the API that I am calling is based on pages with a maximum responses of 500 results per page. I am trying to pull the data since 2017 so its a lot of data.

I want to continue countering until the response is empty.

#!/bin/bash

# Basic while loop
counter=1
for ((i=1;i<=2;i++));
    do
        curl -o gettext.txt --request GET \
        --url "https://api.io/v1/candidates?page=${counter}&per_page=500" \
        --header 'Authorization: Basic aklsjdl;fakj;l;kasdflkaj'
    ((counter++))
    done

echo $counter
echo All done

Anyone have an idea?

Answer 1

As stated in author's comment on his/her own post, the returned data is in json format. The author didn't ask how to append two json files, but it is a necessary step for him/her to accomplish his/her job. In order to append two json's, json1 and json2, maybe skipping json1 last byte } and json2 first byte {, and appending , between them would be enough. Here I am using jq to join two jsons as a more generic approach.

In the examples shown below, the nextjsonchunk file is the json file got at each request. If it has contents, it is appended to the mainjsonfile with jq. If it seems to be empty (inferred by its size) the loop breaks and the result is moved to the present folder and cleanup is made.

Using curl:

#!/usr/bin/env bash

tempfolder=/dev/shm  # temporary memory parition, avaiable in ubuntu
emptyjsonize=10      # the minimum json file length, to be used as a threshold

for ((counter=1; 1; counter++))
do
  curl "https://api.io/v1/candidates?page=${counter}&per_page=500" \
    --header "Authorization: Basic aklsjdl;fakj;l;kasdflkaj" \
    --ouput $tempfolder/nextjsonchunk
  if [ $(wc -c <$tempfolder/nextjsonchunk) -le $emptyjsonize ]; then break; fi
  jq -s '.[0]*.[1]' $tempfolder/mainjsonfile $tempfolder/nextjsonchunk > $folder/mainjsonfile
done
rm $tempfolder/nextjsonchunk # cleaning up
mv $tempfolder/mainjsonfile ./jsonresultfile # end result

Alternativelly, using wget:

#!/usr/bin/env bash

tempfolder=/dev/shm  # temporary memory parition, avaiable in ubuntu
emptyjsonize=10      # the minimum json file length, to be used as a threshold

for ((counter=1; 1; counter++))
do
  wget "https://api.io/v1/candidates?page=${counter}&per_page=500" \
    --header="Authorization: Basic aklsjdl;fakj;l;kasdflkaj" \
    --ouput-document $tempfolder/nextjsonchunk
  if [ $(wc -c <$tempfolder/nextjsonchunk) -le $emptyjsonize ]; then break; fi
  jq -s '.[0]*.[1]' $tempfolder/mainjsonfile $tempfolder/nextjsonchunk > $folder/mainjsonfile
done
rm $tempfolder/nextjsonchunk # cleaning up
mv $tempfolder/mainjsonfile ./jsonresultfile # end result

• It is a good idea to take two sample json and test the merging between them, to check if it is being done properly.

• It is also good to assure if the empty json file check is ok. The 10 byte was just a guess.

• A tmpfs (in memory) partition, /dev/shm was used in the examples, to avoid a many writes, but its use is optional.