Python Subprocess Command as List Not String

Question

I need to use the subprocess module in Python to create some new files by redirecting stdout. I don't want to use shell=True because of the security vulnerabilities.

I wrote some test commands to figure this out, and I found that this worked:

import subprocess as sp
filer = open("testFile.txt", 'w')
sp.call(["ls", "-lh"], stdout=filer)
filer.close()

However, when I passed the command as one long string instead of a list, it couldn't find the file. So when I wrote this:

import subprocess as sp
filer = open("testFile.txt", 'w')
sp.call("ls -lh", stdout=filer)
filer.close()

I received this error:

Traceback (most recent call last):
  File "./testSubprocess.py", line 16, in <module>
    sp.call(command2, stdout=filer)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 524, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1308, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Why does it matter if I pass the arguments as a string or as a list?

Answer 1

It is because of the way how the call happens:

With shell=True, the call is performed via the shell, and the command is given to the shell as one string.

With shell=False, the call is performed directly, via execv() and related functions. These functions expet an array of arguments.

If you only pass one string, it is treated as an abbreviation for a call with only the executable's name without arguments. But there is (probably) no executable called ls -lh on your system.

To be exact, somewhere deep inside subprocess.py, the following happens:

        if isinstance(args, types.StringTypes):
            args = [args]
        else:
            args = list(args)

So every string passed is turned into a list with one element.

        if shell:
            args = ["/bin/sh", "-c"] + args

This one I didn't know: obviously, this allows for passing additional arguments to the called shell. Although it is documented this way, don't use it as it is subject to create too much confusion.

If shell=False, we have further down below

if env is None:
    os.execvp(executable, args)
else:
    os.execvpe(executable, args, env)

which just takes a list and uses it for the call.

Answer 2

If you want your strings to be divided like they would at a shell, use shlex:

import subprocess as sp
import shlex
with open("testFile.txt", 'w') as filer:
    sp.call(shlex.split("ls -lh"), stdout=filer)

BTW, let me make the case for check_call while I'm here. Without it, you'd get empty output if you added an invalid argument, for example. You'd be left wondering why the output at filer is empty.

with open("testFile.txt", 'w') as filer:
    sp.check_call(shlex.split("ls -lh0"), stdout=filer)

With check_call you get an error that localizes the problem and prevents subsequent code from executing:

Traceback (most recent call last):
  File "go.py", line 6, in <module>
    sp.check_call(shlex.split("ls -lh0"), stdout=filer)
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ls', '-lh0']' returned non-zero exit status 2