Pipelining & redirection (Bash)

Uit De Vliegende Brigade
Versie door Jeroen Strompf (overleg | bijdragen) op 3 nov 2022 om 22:27 (→‎See also)
(wijz) ← Oudere versie | Huidige versie (wijz) | Nieuwere versie → (wijz)
Naar navigatie springen Naar zoeken springen

How to use the output of a command as input for another command? An overview:

Execute remotely

This article is about how to use the output of one process as the input to another process. For me, this is pretty much the core of automation. Now to do take this a step further: How to execute stuff on remote servers? To my surprise, this is quite possible over SSH.

MySQL

  • Have a remote MySQL server available locally through a SSH tunnel
  • Execute commands through the usual mysql client

Shell?

Can you execute shell commands on an SSH server as if it was somewhere locally?

Redirect output to a file

Probably most cases in this article are about redirecting the output of a command to somehwere else. This section is about the opposite case: There already is some output, and it's stored in a file. How now to redirect this to somewhere else: 

cat dir.txt

Redirect output to an array

This is pretty cool: Redirect a series of output into an array. If you redirect to a variable, it will become one long string, and not an array.

Parenthesis construct

Quite impressive [1]: 

ab=($(echo a b))

Mapfile

Examples with output from WP-CLI 

mapfile -t j < <( wp post list --post_type=product --field=ID --posts_per_page=64 )

However, the first two entries will be empty: This WP-CLI command incorporates quite some whitespace in its output. Let's fix that: 

mapfile -t j < <( wp post list --post_type=product --field=ID --posts_per_page=64 | grep . )

See section about removing whitespace below, for details.

Examples with SQL (still through WP-CLI) 

i='select term_id from wp_terms join wp_term_taxonomy using (term_id) where taxonomy like "pa_as%"'
mapfile -t j < <( wp db query "$i" --skip-column-names )

echo ${j[@]}
echo "# of entries: ${#j[@]}"
echo ${!j[@]}
echo "Entry 5: ${j[5]}"

I tend to rewrite the mapfile line to mapfile -t j < <( wp db query "$i" --skip-column-names | grep . ), but it doesn't seem necessary here.

Redirect output to a file

As a starter: 

ls >> dir.txt

Redirect output to a variable

What is loosely called redirect in the title of this section, might have a more specific meaning here:

Capture

  • The construct $( ... ) captures the output of a command. Appearantly, that's not the same as a redirect
  • Alternative construct: $(` ... `) - Don't ask me why
  • Alternative construct: ` ... ` - Don't ask me why [2], [3]
  • Alternative construct: "$( ... ) - The double quotes avoids complications with e.g., "*" in the output. Depending on how you handle it, it might expand as a glob
  • This creates a subshell [4]
  • It captures STDOUT, but how about STDERR? [5]

Example: 

dir=$(ls)

[6]: 

... storing output in variables is often unnecessary. For small, short strings
you will need to reference multiple times in your program, this is completely
fine, and exactly the way to go; but for processing any nontrivial amounts of
data, you want to reshape your process into a pipeline, or use a temporary
file

Actual redirect

  • Appearantly, for something that is really called redirection, you would use the command read
  • This approach might still work in situations where capturing might not work
  • It avoids using a subshell

Tee

If I remember correctly, tee is like a T-crossing: Data goes to both the command and to the screen. More here?

Function

[7]:

Using a pipeline getting too complicated? Write a function! Maybe even include the complete pipeline in this function

|, |&, >, >>, >>>, <, <<, <(), < <(), etc.

Haha, this is probably going to be some work in progress...

Some food for thought:

Process substitution - <(), ()>

jq

New: https://stackoverflow.com/questions/63436299/how-iterate-over-wp-options-list-output-using-bash

xargs

xargs is a more explicit way of reusing the output of a command. I often use it, when more implicit ways of pipelining like through > or | doesn't work, like with WP-CLI. See xargs for details.

Trim whitespace around output

A problem I have quite often around redirecting output from WP-CLI: All kind of unwanted output: 

taxid=20
i=20208

# 1 - Not ok
########################################
#
# Output includes CR/LF or something like that
#
echo "	Old name: $(wp --user=4 wc product_attribute_term get $taxid $i --field=name --format=csv)"


# 2 - Not OK
########################################
#
tmp=$(wp --user=4 wc product_attribute_term get $taxid $i --field=name --format=csv)
echo "Old name: $tmp"


# 3 - Works
########################################
#
# * Maybe this is why I use xargs often for processing the output of WP-CLI
#   commands
# * This actually seems to be a legitimate use of xargs. It also reduces
#   multiple consecutive spaces to one:
#   https://stackoverflow.com/questions/369758/how-to-trim-whitespace-from-a-bash-variable
# * However, this doesn't work in combination with redirection - See further below
#
wp --user=4 wc product_attribute_term get $taxid $i --field=name --format=csv | xargs


# 4 - Not OK
########################################
#
wp --user=4 wc product_attribute_term get $taxid $i --field=name --format=ids


# 5 - Not OK
########################################
#
wp --user=4 wc product_attribute_term get $taxid $i --field=id


# 6 - Works
########################################
#
# * Grep seems to filter out CR/LF anyway
# * So I only need a pattern that will accept the string otherwise: grep [0-9a-zA-Z ]
# * In this case: As soon as there is a letter or number in the output, it will be accepted
# * A better solution: "grep ."
# * Contrary to xargs, this still works after redirection
#
wp --user=4 wc product_attribute_term get $taxid $i --field=name --format=csv | grep .

From the two options xarg and grep, only the latter works in combination with further redirecting: 

# Doesn't work
########################################
#
# I think you're not supposed to redirect output of a function like this at all:
# https://unix.stackexchange.com/questions/337291/redirect-command-output-to-variable-bash-script#337292
#
wp --user=4 wc product_attribute_term get $taxid $i --field=name --format=csv | grep . >> tmp   # Doesn't work


# OK - With a intermediate variable
########################################
#
tmp=$(wp --user=4 wc product_attribute_term get $taxid $i --field=name --format=csv | grep .)
echo "Name (1): $tmp"


# OK - Direct
########################################
#
echo "Name (2): $(wp --user=4 wc product_attribute_term get $taxid $i --field=name --format=csv | grep .)"

See also

Sources

Overgenomen van "https://wiki.devliegendebrigade.nl/index.php?title=Pipelining_%26_redirection_(Bash)&oldid=32000"