Linux tee command occasionally fails if followed by a pipe

Question

I run a find command with tee log and xargs process output; by accident I forget add xargs in second pipe and found this question.

The example:

% tree
.
├── a.sh
└── home
    └── localdir
        ├── abc_3
        ├── abc_6
        ├── mydir_1
        ├── mydir_2
        └── mydir_3

7 directories, 1 file

and the content of a.sh is:

% cat a.sh
#!/bin/bash
LOG="/tmp/abc.log"

find home/localdir -name "mydir*" -type d  -print | tee $LOG | echo

If I add the second pipe with some command, such as echo or ls, the write log action would occasionally fail.

These are some examples when I ran ./a.sh many times:

% bash -x ./a.sh; cat /tmp/abc.log  // this tee failed
+ LOG=/tmp/abc.log
+ find home/localdir -name 'mydir*' -type d -print
+ tee /tmp/abc.log
+ echo


% bash -x ./a.sh; cat /tmp/abc.log  // this tee ok
+ LOG=/tmp/abc.log
+ find home/localdir -name 'mydir*' -type d -print
+ tee /tmp/abc.log
+ echo

home/localdir/mydir_2  // this is cat /tmp/abc.log output
home/localdir/mydir_3
home/localdir/mydir_1

Why is it that if I add a second pipe with some command (and forget xargs), the tee command will fail occasionally?

Answer 1

The problem is that, by default, tee exits when a write to a pipe fails. So, consider:

find home/localdir -name "mydir*" -type d  -print | tee $LOG | echo

If echo completes first, the pipe will fail and tee will exit. The timing, though, is imprecise. Every command in the pipeline is in a separate subshell. Also, there are the vagaries of buffering. So, sometimes the log file is written before tee exits and sometimes it isn't.

For clarity, let's consider a simpler pipeline:

$ seq 10 | tee abc.log | true; declare -p PIPESTATUS; cat abc.log
declare -a PIPESTATUS='([0]="0" [1]="0" [2]="0")'
1
2
3
4
5
6
7
8
9
10
$ seq 10 | tee abc.log | true; declare -p PIPESTATUS; cat abc.log
declare -a PIPESTATUS='([0]="0" [1]="141" [2]="0")'
$

In the first execution, each process in the pipeline exits with a success status and the log file is written. In the second execution of the same command, tee fails with exit code 141 and the log file is not written.

I used true in place of echo to illustrate the point that there is nothing special here about echo. The problem exists for any command that follows tee that might reject input.

Documentation

Very recent versions of tee have an option to control the pipe-fail-exit behavior. From man tee from coreutils-8.25:

--output-error[=MODE]
set behavior on write error. See MODE below

The possibilities for MODE are:

MODE determines behavior with write errors on the outputs:

   'warn' diagnose errors writing to any output

   'warn-nopipe'
          diagnose errors writing to any output not a pipe

   'exit' exit on error writing to any output

   'exit-nopipe'
          exit on error writing to any output not a pipe

The default MODE for the -p option is 'warn-nopipe'. The default operation when --output-error is not specified, is to exit immediately on error writing to a pipe, and diagnose errors writing to non pipe outputs.

As you can see, the default behavior is "to exit immediately on error writing to a pipe". Thus, if the attempt to write to the process that follows tee fails before tee wrote the log file, then tee will exit without writing the log file.

Answer 2

Right, piping from tee to something that exits early (not dependent on reading the input from tee in your case) will cause intermittent errors. For a summary of this gotcha see:

http://www.pixelbeat.org/docs/coreutils-gotchas.html#tee