10.5 Shell Substitutions

Contrary to a persistent urban legend, the Bourne shell does not systematically split variables and back-quoted expressions, in particular on the right-hand side of assignments and in the argument of case. For instance, the following code:

     case "$given_srcdir" in
     .)  top_srcdir="`echo "$dots" | sed 's,/$,,'`" ;;
     *)  top_srcdir="$dots$given_srcdir" ;;
     esac

is more readable when written as:

     case $given_srcdir in
     .)  top_srcdir=`echo "$dots" | sed 's,/$,,'` ;;
     *)  top_srcdir=$dots$given_srcdir ;;
     esac

and in fact it is even more portable: in the first case of the first attempt, the computation of top_srcdir is not portable, since not all shells properly understand "`..."..."...`". Worse yet, not all shells understand "`...\"...\"...`" the same way. There is just no portable way to use double-quoted strings inside double-quoted back-quoted expressions (pfew!).

$@

One of the most famous shell-portability issues is related to ‘"$@"’. When there are no positional arguments, Posix says that ‘"$@"’ is supposed to be equivalent to nothing, but the original Unix version 7 Bourne shell treated it as equivalent to ‘""’ instead, and this behavior survives in later implementations like Digital Unix 5.0.

The traditional way to work around this portability problem is to use ‘${1+"$@"}’. Unfortunately this method does not work with Zsh (3.x and 4.x), which is used on Mac OS X. When emulating the Bourne shell, Zsh performs word splitting on ‘${1+"$@"}’:

          zsh $ emulate sh
          zsh $ for i in "$@"; do echo $i; done
          Hello World
          !
          zsh $ for i in ${1+"$@"}; do echo $i; done
          Hello
          World
          !
     

Zsh handles plain ‘"$@"’ properly, but we can't use plain ‘"$@"’ because of the portability problems mentioned above. One workaround relies on Zsh's “global aliases” to convert ‘${1+"$@"}’ into ‘"$@"’ by itself:

          test "${ZSH_VERSION+set}" = set && alias -g '${1+"$@"}'='"$@"'
     

A more conservative workaround is to avoid ‘"$@"’ if it is possible that there may be no positional arguments. For example, instead of:

          cat conftest.c "$@"
     

you can use this instead:

          case $# in
          0) cat conftest.c;;
          *) cat conftest.c "$@";;
          esac
     

Autoconf macros often use the set command to update ‘$@’, so if you are writing shell code intended for configure you should not assume that the value of ‘$@’ persists for any length of time.

${10}

The 10th, 11th, ... positional parameters can be accessed only after a shift. The 7th Edition shell reported an error if given ${10}, and Solaris 10 /bin/sh still acts that way:

          $ set 1 2 3 4 5 6 7 8 9 10
          $ echo ${10}
          bad substitution
     

${var:-value}

Old BSD shells, including the Ultrix sh, don't accept the colon for any shell substitution, and complain and die.

${var=literal}

Be sure to quote:

          : ${var='Some words'}
     

otherwise some shells, such as on Digital Unix V 5.0, die because of a “bad substitution”.

Solaris /bin/sh has a frightening bug in its interpretation of this. Imagine you need set a variable to a string containing ‘}’. This ‘}’ character confuses Solaris /bin/sh when the affected variable was already set. This bug can be exercised by running:

          $ unset foo
          $ foo=${foo='}'}
          $ echo $foo
          }
          $ foo=${foo='}'   # no error; this hints to what the bug is
          $ echo $foo
          }
          $ foo=${foo='}'}
          $ echo $foo
          }}
           ^ ugh!
     

It seems that ‘}’ is interpreted as matching ‘${’, even though it is enclosed in single quotes. The problem doesn't happen using double quotes.

${var=expanded-value}

On Ultrix, running

          default="yu,yaa"
          : ${var="$default"}
     

sets var to ‘M-yM-uM-,M-yM-aM-a’, i.e., the 8th bit of each char is set. You don't observe the phenomenon using a simple ‘echo $var’ since apparently the shell resets the 8th bit when it expands $var. Here are two means to make this shell confess its sins:

          $ cat -v <<EOF
          $var
          EOF
     

and

          $ set | grep '^var=' | cat -v
     

One classic incarnation of this bug is:

          default="a b c"
          : ${list="$default"}
          for c in $list; do
            echo $c
          done
     

You'll get ‘a b c’ on a single line. Why? Because there are no spaces in ‘$list’: there are ‘M- ’, i.e., spaces with the 8th bit set, hence no IFS splitting is performed!!!

One piece of good news is that Ultrix works fine with ‘: ${list=$default}’; i.e., if you don't quote. The bad news is then that QNX 4.25 then sets list to the last item of default!

The portable way out consists in using a double assignment, to switch the 8th bit twice on Ultrix:

          list=${list="$default"}
     

...but beware of the ‘}’ bug from Solaris (see above). For safety, use:

          test "${var+set}" = set || var={value}
     

`commands`

Posix requires shells to trim all trailing newlines from command output before substituting it, so assignments like ‘dir=`echo "$file" | tr a A`’ do not work as expected if ‘$file’ ends in a newline.

While in general it makes no sense, do not substitute a single builtin with side effects, because Ash 0.2, trying to optimize, does not fork a subshell to perform the command.

For instance, if you wanted to check that cd is silent, do not use ‘test -z "`cd /`"’ because the following can happen:

          $ pwd
          /tmp
          $ test -z "`cd /`" && pwd
          /
     

The result of ‘foo=`exit 1`’ is left as an exercise to the reader.

The MSYS shell leaves a stray byte in the expansion of a double-quoted command substitution of a native program, if the end of the substution is not aligned with the end of the double quote. This may be worked around by inserting another pair of quotes:

          $ echo "`printf 'foo\r\n'` bar" > broken
          $ echo "`printf 'foo\r\n'`"" bar" | cmp - broken
          - broken differ: char 4, line 1
     

$(commands)

This construct is meant to replace ‘`commands`’, and it has most of the problems listed under `commands`.

This construct can be nested while this is impossible to do portably with back quotes. Unfortunately it is not yet universally supported. Most notably, even recent releases of Solaris don't support it:

          $ showrev -c /bin/sh | grep version
          Command version: SunOS 5.10 Generic 121004-01 Oct 2005
          $ echo $(echo blah)
          syntax error: `(' unexpected
     

nor does irix 6.5's Bourne shell:

          $ uname -a
          IRIX firebird-image 6.5 07151432 IP22
          $ echo $(echo blah)
          $(echo blah)
     

If you do use ‘$(commands)’, make sure that the commands do not start with a parenthesis, as that would cause confusion with a different notation ‘$((expression))’ that in modern shells is an arithmetic expression not a command. To avoid the confusion, insert a space between the two opening parentheses.

Avoid commands that contain unbalanced parentheses in here-documents, comments, or case statement patterns, as many shells mishandle them. For example, Bash 3.1, ‘ksh88’, pdksh 5.2.14, and Zsh 4.2.6 all mishandle the following valid command:

          echo $(case x in x) echo hello;; esac)
     

^

Always quote ‘^’, otherwise traditional shells such as /bin/sh on Solaris 10 treat this like ‘|’.