sed: couldn't write 26 items to stdout: Broken pipe

Question

i have the following command:

cat input.txt | awk '{print $1, $6}' | sort -n | uniq -c | sort -nr | sed 's/"//g'| head -10

i get the desired output, but i get this error

sed: couldn't write 26 items to stdout: Broken pipe

where input.txt is something like:

192.168.2.20 - - [28/Jul/2006:10:27:10 -0300] "GET /cgi-bin/try/ HTTP/1.0" 200 3395
127.0.0.1 - - [28/Jul/2006:10:22:04 -0300] "GET / HTTP/1.0" 200 2216

what am i missing

Answer 1

As @KamilCuk said in a comment, this is happening because head -10 only reads the first 10 lines from the pipeline (plus maybe some input buffering), and then closes it; if the input is big enough, this happens before sed has written everything into the pipe (and the pipe's buffer isn't big enough to absorb the extra). So whether this happens or not depends on the input size, OS and its parameters (which determine the pipe's characteristics), sed's behavior on having its output dropped, etc. Just changing things up a bit may be enough to avoid the problem, for example:

...sort -nr | tr -d '"' | head -10       # use `tr` instead of `sed` -- it may behave differently
...sort -nr | head -10 | sed 's/"//g'    # swap `head` and `sed` -- now `sort`'s output is dropped

And here's one that will avoid the error:

...sort -nr | sed '11,$ d; s/"//g'

The way this works is it tells sed to discard lines 11 through the end of input ("$"), but since it discards them after reading them (rather than never reading them in the first place, like head -10), sort's entire output gets read and no error occurs.

BTW, as @triplee pointed out, using cat at the beginning of the pipeline is useless; you should have awk read the file directly, like this:

awk '{print $1, $6}' input.txt | ...