Multiple commands (GNU Parallel)

How to include multiple commands in a GNU Parallel statement?

Use a script file

parallel < my_script.sh

No difference between operators & arguments - Example Leo

[1]:

$ parallel "{1} {2}" ::: 'printf "%02d "' 'printf "%03d "' ::: 1 2
01 02 001 002

What it does:

printf "%02d ": Print "00 "
printf "%03d ": Print "000 "

Arguments are multiplied into:

* printf "%02d " 1
* printf "%02d " 2
* printf "%03d " 1
* printf "%03d " 2

and this is executed through parallel.

Small detail: I get output as above, but when I run the print commands separately, I get more '0's. Maybe has to do with the single quotes around the print statements?

Use a function

From Chapter 5 of the manual:

The command can be a script, a binary or a Bash function if the function
is exported using

   export -f :

my_func()
{
   echo in my_func $1
}

export -f my_func

parallel my_func ::: 1 2 3

Note export -f: Parallel operates within a subshell, and stuff from the invoking shell has to be made available in the subshell, if needed. See elswhere in the article for details.

Syntaxis for subshell?

The command to be executed in parallel, have to be enveloped with the right syntaxis, to be executed as one unit (subshell?) at the right moment:

Appearantly, use $(cmd1; cmd2; cmd3) to include multiple commands in a pipeline [2]
I have better experiences with using backticks for this.

Example SO

seq 5 | parallel echo '$(j=$((2*{})); echo $(($j+100)))'

It looks a bit weird to have two echo commands, but it works

More detailed

It seems I have to use apostrophes, rather than $(). E.g., the following statements all work:

seq 5 | parallel echo '$(j=$((2*{})); echo $(($j+100)))'
parallel echo '$(j=$((2*{})); echo $(($j+100)))' ::: $(seq 5)
parallel 'echo "Parallel: "; echo {1} {2} {3}' ::: $(seq 5)
parallel 'echo "Parallel: "; echo {1} {2} {3}' ::: $(seq 5)

But nothing works if I try to envelop stuff in $(). E.g.:

$ parallel $(echo {}; echo {}) ::: $(seq 3)

/bin/bash: 1: command not found
/bin/bash: 2: command not found
/bin/bash: 3: command not found

But this works:

$ parallel 'echo {}; echo {}' ::: $(seq 3)

1
1
2
2
3
3

Multiple commands inline - Example

With the right syntaxis, it's perfectly possible to include multiple statements when invoking GNU Parallel.

Let's start here:

parallel echo ::: $(seq 5)

which is synonymous to

parallel echo ::: `seq 5`

And these two commands already hold the key to executing multiple commands in a parallel invocation: You have to encapsulate them, so that they get executed as one unit (subshell?) at the right moment.

Now expand this to two commands with some trial and error:

# I guess parallel doesn't see a connection between the echo statement
# and the seq statement
#
$ parallel echo "foobar"; echo {} ::: `seq 5`

parallel: Warning: Input is read from the terminal. You either know what you
parallel: Warning: are doing (in which case: YOU ARE AWESOME!) or you forgot
parallel: Warning: ::: or :::: or to pipe data into parallel. If so
parallel: Warning: consider going through the tutorial: man parallel_tutorial
parallel: Warning: Press CTRL-D to exit.

Another trial with error:

# The part between $() gets evaluated first and results in "foobar 1",
# "foobar 2", etc. - lacking bash commands
#
$ parallel $(echo "foobar"; echo {}) ::: `seq 5`

/bin/bash: foobar: command not found
/bin/bash: foobar: command not found
/bin/bash: foobar: command not found
/bin/bash: foobar: command not found
/bin/bash: foobar: command not found

Finally, a trial-without-error:

$ parallel echo $(echo "foobar"; echo {}) ::: `seq 5`

foobar 1
foobar 2
foobar 3
foobar 4
foobar 5

It may look like there is seemingly one echo statement too much, but it really isn't: The once between $() get evaluated and therefore disappear before Parallel gets to to anything - And Parallel does need a command to execute
Note that the stuff between $() does get the 'distributed input'

Let's expand this with a static additional statement, just for size:

parallel echo $(echo "Hoi"; echo {}; echo $((1+1))) ::: $(seq 5)

Let's now make the sum in the last statement dynamic:

$ parallel echo $(echo "Hoi"; echo {}; echo $(({}+{}))) ::: $(seq 5)

bash: {}+{}: syntax error: operand expected (error token is "{}+{}")
...

The problem: The inner stuff gets evaluated before {} gets substituted by Parallel.

Change this by putting the stuff that needs to be substituted through Parallel first, between apostrophes:

$ parallel$ parallel echo '$(echo "Hoi"; echo {}; echo $(({}+{})))' ::: $(seq 5)

Hoi 1 2
Hoi 2 4
Hoi 3 6
Hoi 4 8
Hoi 5 10

How I tend to interpret this:

$() is general Bash syntaxis concerning redirecting (?) the outcome of something
The apostrophes are a Parallel trick, to change the order of evaluation.

In this last example, let's see what happens if you remove the $() part:

$ parallel echo 'echo "Hoi"; echo {}; echo $(({}+{}))' ::: $(seq 5)
echo Hoi
1
2
echo Hoi
2
4
echo Hoi
3
6
echo Hoi
4
8
echo Hoi
5
10

What I think is happening:

There no part that gets evaluted first. It's just like calling parallel with something like

parallel echo ::: (echo "hoi"; echo {}; echo $(({}+{}))) ::: $(seq 5)

as if the first part is just an array with 3 elements. Note that {} does get substituted correctly.

Splitting over multiple lines

Let's try to split the inline commands over multiple lines, to make them more readible:

parallel echo \
$(  \
   echo "Foo"; \
   echo "bar"; \
   echo "One"; \
   echo {} \
) ::: $(seq 5)

Output:

Foo bar One 1
Foo bar One 2
Foo bar One 3
Foo bar One 4
Foo bar One 5

This works too:

parallel echo \
$(  \
   echo "Foo"; \
   echo "bar"; \
   echo "One"; \
   echo {} \
) \
::: $(seq 5)

Hierarchie of ()

This doesn't work: Bash gets confused by the ()'s:

parallel echo \
$( \
   sql="update wp_terms join wp_term_taxonomy using (term_id) "; \
   sql+="set slug=replace(slug, '{1}', '{2}') "; \
   sql+="where taxonomy='product_cat';"; \
   echo "$sql" \
) ::: $(seq 5) :::+ $(seq 6 10)