Welcome to End Point’s blog

Ongoing observations by End Point people

Find your Perl in Other Shells

Often when programming, it turns out the best tools for the job are system tools, even in an excellent language like Perl. Perl makes this easy with a number of ways you can allocate work to the underlying system: backtick quotes, qx(), system(), exec(), and open(). Virtually anyone familiar with Perl is familiar with most or all of these ways of executing system commands.

What's perhaps less familiar, and a bit more subtle, is what Perl really does when handing these off to the underlying system to execute. The docs for exec() tell us the following:

       exec LIST
       exec PROGRAM LIST
            If there is more than one argument in LIST, or if LIST is an
            array with more than one value, calls execvp(3) with the
            arguments in LIST.  If there is only one scalar argument or an
            array with one element in it, the argument is checked for shell
            metacharacters, and if there are any, the entire argument is
            passed to the system's command shell for parsing (this is
            "/bin/sh -c" on Unix platforms, but varies on other platforms).

That last parenthetical is a key element when we "shell out" and expect certain behavior. Perl is going to use /bin/sh. But I don't; I use /bin/bash. And I am very happy to ignore this divergence ... until I'm not.

Without considering any of these issues, I had leveraged a shell command to do a nifty table comparison between supposedly replicated databases to find where the replication had failed. The code in question was the following:

$ diff -u \
> <(psql -c "COPY (SELECT * FROM foo ORDER BY foo_id) TO STDOUT" -U chung chung) \
> <(psql -c "COPY (SELECT * FROM foo ORDER BY foo_id) TO STDOUT" -U chung chung2)

The above code produced exactly the results I was looking for, so I integrated it into my Perl code via qx() and ran it. Doing so produced the following surprising result:

$ ./
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `diff -u <(psql -c "COPY (SELECT * FROM foo ORDER BY foo_id) TO STDOUT" -U chung chung) <(psql -c "COPY (SELECT * FROM foo ORDER BY foo_id) TO STDOUT" -U chung chung2)'

I worked with an End Point colleague and figured out the problem. <() is supported by bash, but not by Bourne. Further, there is no way to instruct Perl to use /bin/bash for its target shell.

In order to access a different shell and leverage the desired features, I had to use the invocation described for PROGRAM LIST, but identify the shell itself as my program. While I'm unaware of any way to accomplish this with backticks, I was certainly able to do so using Perl's open():

my $cmd = q{diff -u <(psql -c "COPY (SELECT * FROM foo ORDER BY foo_id) TO STDOUT" -U chung chung) <(psql -c "COPY (SELECT * FROM foo ORDER BY foo_id) TO STDOUT" -U chung chung2)};

    open (my $pipe, '-|', '/bin/bash', '-c', $cmd)
        or die "Error opening pipe from diff: $!";
    while (<$pipe>) {
        # do my stuff
    close ($pipe);

Now results between command line and invocation from Perl are consistent. And now I understand for future reference how to control which shell I find my Perl in.

1 comment:

Jon Jensen said...

All of the above is tricky, tricky enough if you're not familiar with it, but then there's the further question of why /bin/sh isn't behaving like bash when it *is* bash on many modern Linux systems, such as RHEL, CentOS, and Fedora:

% ls -lFa /bin/bash /bin/sh
-rwxr-xr-x. 1 root root 967008 Nov 27 07:07 /bin/bash*
lrwxrwxrwx. 1 root root 4 Dec 13 22:49 /bin/sh -> bash*

Like some other common tools such as mailq/newaliases/sendmail, ex/view/rvi/rview/vi (from Vim), slogin/ssh, bzcat/bunzip2, bash looks at the name it was invoked by and behaves differently depending on what it sees.

From the bash manpage:

If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible, while conforming to the POSIX standard as well. [snip more details]

But note that the symlink from /bin/sh -> /bin/bash is common but not any universal standard. A while back Debian switched it to dash, a lighter-weight shell, to speed up boot times and reduce memory usage. Doing so caused a lot of breakage at first as many shell scripts pointed to /bin/sh but were using bash-specific features and had to be changed to be Bourne shell compatible or else point to /bin/bash. OpenBSD defaults to /bin/ksh for its /bin/sh. And I think the other BSDs don't use bash by default either.

So it's probably a blessing that bash invoked as /bin/sh behaves as if it were a simpler Bourne shell, to avoid luring us into the platform-specific false assumption that /bin/sh is always bash.