Why does "pip list" not like broken pipes on the bash command line?

Question

I'm using a bash terminal.

I just wanted to peek at the first few names of installed python packages:

pip list | head -n5

And got an error message:

Package            Version
------------------ ----------------
attrs              21.2.0
Automat            20.2.0
Babel              2.8.0
ERROR: Pipe to stdout was broken
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe

Is something broken? If I'm going to capture that information programmatically, I definitely don't want an error message included.

Answer 1

This might not completely answer your question, but realize that that error message is Python accurately detecting an abrupt break in the pipe consuming the standard out from pip list, in this case just after the fifth line.

As is common with many programs, data comes out in two different streams: standard out (stdout) and standard error (stderr). Since you are at the terminal, you see both as text to the screen. The desired output (i.e., the "Package" and "Version" information) is streaming to the standard output while the error message is actually going to standard error.

If you did want to capture the standard output to a file (via the > redirect), that error message would still end up on the screen instead of in the file:

pip list | head -n5 > pip_list.txt
ERROR: Pipe to stdout was broken
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe

You could get a similar list via pip freeze, which doesn't seem to complain when a pipe quits pulling information from it:

pip freeze | head -n5
attrs==21.2.0
Automat==20.2.0
Babel==2.8.0
bcrypt==3.2.0
blinker==1.4

Since pip freeze can be used to make a requirements.txt file, it may be closer to what you want anyway. Also, unlike pip list you don't have the two-line header that you might want to skip.

If you really really do want to use pip list but not get that error message, you could just redirect stderr to /dev/null (a virtual nowhere, a black hole) via the 2> redirect:

pip list 2> /dev/null | head -n5
Package            Version
------------------ ----------------
attrs              21.2.0
Automat            20.2.0
Babel              2.8.0

Answer 2

Is something broken?

Not in the sense you mean.

The head command terminates after it copies the the specified number of initial lines from its input to its output. This is normal and expected.

If the input is coming from another command via a pipe, and that command attempts to write additional data when there is no longer anything on the consuming end of the pipe, then the system reports an error to it. The conventional terminology for this kind of error is a "broken pipe". This also is normal and expected. It is an error from the perspective of the program receiving it because it prevents the program from completing writing its output.

It is up to the program receiving such an error what to do with it. When that program is a Python interpreter, it responds by throwing a BrokenPipeError, which the Python program has the opportunity to handle in a way appropriate to it. Oftentimes the most appropriate thing to do with that or another OSError is nothing, so that it rises all the way to the top level and causes Python to terminate. That is what pip has done.

If I'm going to capture that information programmatically, I definitely don't want an error message included.

This is why the system provides two standard streams for output:

• the "standard output", intended for programs to use for normal output, and
• the "standard error", intended for programs to use for diagnostic messages and similar data that are somehow different in nature from the data emitted to the standard output.

Programs are in control of what data they write to each, but you can generally expect that they will write error messages to standard error. The Python interpreter indeed does this. When you run a program interactively in the terminal, both standard output and standard error default to writing data to the terminal, but they can be captured or redirected independently. That's one of the main reasons for maintaining a separation in the first place.