Sending command output to a unique file using xargs in Bash

Question

I am trying to run a number of processes dynamically on a fixed number of processors. I want to print the output to a unique file for each process but there is some problem with xargs not using the in-place filename to create a separate file for each process.

The Bash script calls a Csh script and is below:

$ cat temp | xargs -P 8 % csh '%'.csh >& '%'.log

Where temp is a text file of csh command names.

My problem is that xargs takes %.log literally and constantly overwrites the file as the processes write to it, rather than having seperate .log files as desired.

I run this script as $ bash run.bash &

Answer 1

In general, using string replacement to substitute into code is a Bad Idea -- in your case, if you had a script with a malicious name, that name could be used to run arbitrary commands. (Sure, you're executing the script, but the same would apply in situations where you were purely dealing with data and output file names -- so it's best to make a habit of the robust approach regardless).

Pass the names as parameters to the script, rather than substituting them into the script (as xargs would be doing if you fixed its usage by adding -I or -J parameters:

# best-practice approach: run a completely fixed shell script, passing arguments on
# its command line.
xargs -P 8 -n 1 \
  sh -c 'for x; do csh "${x}.csh" >"${x}.log" 2>&1; done' _

You'll note that there's a sh -c instance invoked: This is needed because xargs itself doesn't understand shell operations such as redirections; if you want a redirection to be performed, you need a shell to do it.

---

Now, let's go a little more into why your original code behaved as it did:

xargs -P 8 % csh '%'.csh >& '%'.log

...first performs the redirection to %.log, then runs the command

xargs -P 8 % csh '%'.csh

There's no opportunity for xargs to replace the %.log string, because that redirection was performed by the enclosing shell before the xargs command was run at all.

Answer 2

Using GNU Parallel it looks like this:

cat temp | parallel -P 8 'csh {}.csh >& {}.log'

If you have 8 cores you can even do:

cat temp | parallel 'csh {}.csh >& {}.log'

GNU Parallel quotes {} so that malicious input is will not be executed.