GNU parallel with two arguments

Question

I have a C-shell script with a variable called $hosts_string of the form:

host1,host2,...,hostN

I also have a variable called $chrs_string of the form:

chr1,chr2,...,chrM

I also have a variable called $inputFn which specifies a text file that I need to process.

What I'd like to do is have each host run a command called doStuff on each member of $chrs_string, taking $inputFn as the input filename argument.

I tried the following:

parallel -S $hosts_string 'doStuff {1} {2} > {1}' ::: {$chrs_string} ::: {$inputFn}

If this works, this should create files chr1 through chrM as output from the command doStuff, but I get nothing. Parallel completes without issuing an error, but there is no output.

The command itself works fine if I don't use parallel, so it's not the command, but how I am specifying arguments to parallel.

To confirm this, I tried the following:

parallel -S $hosts_string 'echo {1} {2} > {1}' ::: {$chrs_string} ::: {$inputFn}

If this were to work, and $inputFn is (for example) /foo/bar, then I would get (for example) a file called chr1, containing the string:

chr1 /foo/bar

How do I adjust my parallel command so that it loops over each combination of $chrs_string and $inputFn correctly?

Answer 1

You are close. You just forgot ::: is not the same as ::::

parallel -S $hosts_string 'echo {1} {2} > {1}' ::: {$chrs_string} :::: {$inputFn}

If you just want the filename in the echo you simply do:

parallel -S $hosts_string echo '{1}' {$inputFn} '> {1}' ::: {$chrs_string}