Direct STDERR when opening pipe in perl

Question

I am using open( my $command_out, "-|", $command_string ) to execute a command and process its output on the fly (not having to wait for the command to finish first, as in system()).

I noticed that when I call some R scripts this way, some of R messages are printed to the screen (e.g. Loading required package: ...). I guess this is because R sends this output to stderr (? although these are not really errors).

Is it possible to direct this output too to $command_out when open()-ing so the screen will remain clean?

Answer 1

Say your R program is

#! /usr/bin/env r

require(Hmisc)

cat(argv[1], "\n")

which turns out to be surprisingly chatty:

$ ./prog.r foo
Loading required package: Hmisc
Loading required package: methods
Loading required package: survival
Loading required package: stats
Loading required package: utils
Loading required package: graphics
Loading required package: splines

Attaching package: 'Hmisc'


    The following object(s) are masked from package:survival :

     untangle.specials 


    The following object(s) are masked from package:base :

     format.pval,
     round.POSIXt,
     trunc.POSIXt,
     units 

foo

From here, you could discard the standard error or merge it with another stream.

Discarding the standard error

One way to discard the standard error is to use a shell redirect of 2>/dev/null. This is a general mechanism, and 2 is the standard error's file descriptor. For example:

#! /usr/bin/perl

use warnings;
use strict;

open my $command_out, "-|", "./prog.r foo 2>/dev/null"
  or die "$0: could not start prog.r";

while (<$command_out>) {
  print "got: $_";
}

The shell will also process backtick or qx// expressions

#! /usr/bin/perl

use warnings;
use strict;

(my $command_out = `./prog.r foo 2>/dev/null`) =~ s/^/got: /mg;
print $command_out;

and a scalar command passed to system

#! /usr/bin/perl

use warnings;
use strict;

system("./prog.r foo 2>/dev/null") == 0
  or warn "$0: prog.r exited " . ($? >>8);

For all of these, the output is

$ ./prog.pl 
got: foo

But sometimes you don't want the shell putting its grubby mitts on your command. Maybe it contains shell metacharacters or quotes that you don't want to deal with escaping. This is when a safe pipe-open is useful:

#! /usr/bin/perl

use warnings;
use strict;

my $pid = open my $command_out, "-|";
die "$0: fork: $!" unless defined $pid;

if ($pid == 0) {
  # child
  open STDERR, ">", "/dev/null" or die "$0: open: $!";
  exec "./prog.r", "foo & bar"  or exit 1;  # STDERR silent now
}

while (<$command_out>) {
  print "got: $_";
}

close $command_out or warn "$0: prog.r exited " . ($? >> 8);

Opening a handle on "-|" forks a child and connects the child's standard output to that handle. Like fork, it returns 0 to the child, a non-zero process identifier to the parent, or the undefined value on failure.

In the child, we first redirect STDERR to /dev/null and then use exec to replace the child with our R program. Note that we passed the command in list form to bypass the shell:

If there is more than one argument in LIST, or if LIST is an array with more than one value, calls execvp(3) with the arguments in LIST.

Because we can't see the standard error any longer, it's important to explicitly close $command_out to check that the child ran happily. Otherwise, you'd get a puzzling silent failure.

Sample run:

$ ./prog.pl 
got: foo & bar

Merge STDERR into STDOUT

To see the standard error on your handle, use 2>&1 instead, e.g.,

open my $command_out, "-|", "./prog.r foo 2>&1" or die;

With a safe pipe-open, dup the standard error onto the standard output:

if ($pid == 0) {
  # child
  open STDERR, ">&", \*STDOUT  or die "$0: open: $!";
  exec "./prog.r", "foo & bar" or die "$0: exec: $!";
}

The open documentation covers this:

You may also, in the Bourne shell tradition, specify an EXPR beginning with >&, in which case the rest of the string is interpreted as the name of a filehandle (or file descriptor, if numeric) to be duped (as dup(2)) and opened.

Even though you can see the standard error this way, it's still a good idea to check the child's exit status with close.

Now everything arrives over $command_out:

got: Loading required package: Hmisc
got: Loading required package: methods
got: Loading required package: survival
got: Loading required package: stats
got: Loading required package: utils
got: Loading required package: graphics
got: Loading required package: splines
got: 
got: Attaching package: 'Hmisc'
got: 
got: 
got:    The following object(s) are masked from package:survival :
got: 
got:     untangle.specials 
got: 
got: 
got:    The following object(s) are masked from package:base :
got: 
got:     format.pval,
got:     round.POSIXt,
got:     trunc.POSIXt,
got:     units 
got: 
got: foo & bar