How can I control the number of running processes?

Question

I'm doing a shell script and the unknown situation occurred. I must execute a certain number of processes, lets suppose 12. But I want to limit the executions by 't' each time. So if 't' is 3 I would have to execute 3 processes then 3, 3 and finally 3. But I would like to do it automatically, so I need to monitore those running processes and when one of them has finished I must execute one of the remaining processes.

After some research, I have found the following command:

launch backgroundprocess &
PROC_ID=$!
while kill -0 "$PROC_ID" >/dev/null 2>&1; do
    echo "PROCESS IS RUNNING"
done
echo "PROCESS TERMINATED"

Proposed by cuonglm.

This can help to know if a process is running or not. I tried to create 12 processes and save them in 3 different variables, but it isn't working properly.

processors=3
counter=0

for A in {1..12}
do
    counter=$((counter+1))
    backgroundprocess &
    PID[$A]=$!

    while [ $counter -eq $processors ]
    do
        if kill -0 "$PID[1]" >/dev/null 2>&1;
        then
            counter=$(($counter-1))
            break
        fi
    done
done

Do any of you know how can I do this work?

Answer 1

The jobs built-in command of the shell can be used to count the background processes.

An additional file is used to wait the termination of background processes.

Give it a try, the tested script is below:

#!/bin/bash --

tmp_file=/tmp/`date "+%Y%m%d%H%M%S$$"`
rm -f "${tmp_file}"
touch "${tmp_file}"
max_nb_processes=12
max_parallel_nb_processes=3
nb_processes=0
while [ $nb_processes -lt $max_nb_processes ]
do
  if [ `jobs -r | wc -l` -lt $max_parallel_nb_processes ]
  then
    (backgroundprocess ; printf "end" "" >> "${tmp_file}")&
    ((nb_processes ++))
  else
    read -t 10 line < "${tmp_file}"
  fi
done
wait

---

For reference, the first version.

The tested version below use some polling:

#!/bin/bash --

poll_time_second=10
max_nb_processes=12
max_parallel_nb_processes=3
nb_processes=0
while [ $nb_processes -lt $max_nb_processes ]
do
  if [ `jobs -r | wc -l` -lt $max_parallel_nb_processes ]
  then
    backgroundprocess &
    ((nb_processes ++))
  else
    sleep $poll_time_second
  fi
done
wait

As long as there is less than 3 background processes, a new process is started in background.

When there are 3 background processes, the script sleep 10 seconds, before checking again.

When 12 background processes had been started, the script wait that the last ones end before terminating.