return a variable value from a subprocess in python

Question

I have a code workflow in which from a main script(level 0) I call another script through subprocess. This subprocess script (level 1) in turn calls another script as a subprocess. Now from this level 2 subprocess script I want to return the value of a variable upto the main script(level 0). I have tried Popen.communicate() but I am not able to return the value. My current code is as:

main_script.py

def func():

    para = ['a','b']
    result = subprocess.Popen([sys.executable,"first_subprocess.py"]+para,stdout=subprocess.PIPE)
    result.wait()

    return_code = result.returncode
    out, err = sub_result.communicate()


    if return_code == 1:
        return return_code

    else:
        if out is not None:
            print 'return value: ', out


if __name__ == '__main__':

    func()

From above script is called first_subprocess.py which has:

def some_func():

    # some other code and multiple print statements

    para = ['a','b']
    result = subprocess.Popen([sys.executable,"second_subprocess.py"]+para,stdout=subprocess.PIPE)

    result.wait()
    out, err = result.communicate()
    return_code = sub_result.returncode
    if return_code == 0:
        return out


if __name__ == '__main__':

    some_func()

The second_subprocess.py returns a value like:

def test():
    # some other code and multiple print statements
    val = 'time'
    print 'returning value'
    return val   

if __name__ == '__main__':    

    test()

When I try above code I get all the print statements in the codes as an output but not the return value. Even if try to print the variable value in subprocess instead of doing a return it wont serve the purpose because there are multiple print statements.

How can I return the variable value in this case?

UPDATED VERSION:

After @Anthons suggestion I have modifed my first_subprocess.py script and main_script.py as follows:

first_subprocess.py:

def some_func():

   try:
    key = None
    if not (key is None):

       para = ['a','b']
       result = subprocess.Popen([sys.executable,"second_subprocess.py"]+para,stdout=subprocess.PIPE)

       sub_result.wait()
       out, err = sub_result.communicate()
       return_code = sub_result.returncode
       if return_code == 0:
       for line in out.splitlines():
           if not line.startswith('>>>'):
              continue
           print line
   else:
     sys.exit(0)
 except:
   return 1

Main_script.py:

if out is not None:
   for line in out.splitlines():
       if not line.startswith('>>>'):
          continue
      value = line.split(':',1)[1].lstrip()

print 'return value:',value`

When I execute above I get UnboundLocalError: local variable 'value' referenced before assignment at the print value command. It seems if I do not execute the code in level 1 script and do a sys.exit() then out in main script is neither empty nor none but it has some undefined value and thus the value variable doesn't get initialized and throws error

Answer 1

If you just want to return an integer value you can use the exit value. This is not the same a returning from some_func(), you would have to do sys.exit(integer_val).

If you want to return a string like time you should print that ( or write to sys.stdout ) and then in the calling process (level 1) parse the output from the subprocess and print it to its own stdout for level 0 to see it.

In your case level two should do something like:

def test():
    # some other code and multiple print statements
    val = 'time'
    print 'returning value:', val

if __name__ == '__main__':    
    test()

And at level 1 you would do:

def some_func():

    # some other code and multiple print statements

    para = ['a','b']
    result = subprocess.Popen([sys.executable,"second_subprocess.py"]+para,stdout=subprocess.PIPE)

    result.wait()
    out, err = result.communicate()
    return_code = sub_result.returncode
    if return_code == 0:
        print out

if __name__ == '__main__':
    some_func()

With that main_script.py has something to read from the invocation of your level 1 script.

I normally use subprocess.check_output() for these kind of passing on info. That throws an exception if the called process has a non-zero exit status (i.e. on error). I can also recommend that if the subprocess writes more info than just the variable, you make the output lines easily parseable by returning something unique at the beginning of the line (so you still can use print statements for debugging the individual scripts and get the right value from the output):

Level 2:

def test():
    # some other code and multiple print statements
    print 'debug: Still going strong'
    val = 'time'
    print '>>>> returning value:', val

if __name__ == '__main__':    
    test()

Level 1:

...
out, err = result.communicate()
for line in out.splitlines():
    if not line.startswith('>>>>'):
        continue
    print line
...

Level 0:

...
out, err = result.communicate()
for line in out.splitlines():
    if not line.startswith('>>>>'):
        continue
    try:
        value = line.split(':', 1)[1]
    except IndexError:
        print 'wrong input line', repr(line)
    print 'return value: ', value
...

---

The following files work together. Save them under their indicated names

lvl2.py:

# lvl2
import sys

def test():
    # some other code and multiple print statements
    print >> sys.stderr, 'argv', sys.argv[1:]
    print 'debug: Still going strong'
    val = 'time'
    print '>>>> returning value:', val
    return 0

if __name__ == '__main__':
    sys.exit(test())

lvl1.py:

# lvl1.py
import sys
import subprocess

def some_func():
    para = ['a','b']
    sub_result = subprocess.Popen(
        [sys.executable, "lvl2.py" ] + para,
        stdout=subprocess.PIPE)
    sub_result.wait()
    out, err = sub_result.communicate()
    return_code = sub_result.returncode
    if return_code == 0:
        for line in out.splitlines():
            if not line.startswith('>>>'):
                continue
            print line
    else:
        print >> sys.stderr, 'level 2 exited with' + return_code
    sys.exit(0)

if __name__ == '__main__':
    sys.exit(some_func())

lvl0.py:

# lvl0
import subprocess
import sys

def func():
    para = ['a','b']
    result = subprocess.Popen(
        [sys.executable, "lvl1.py"] + para,
        stdout=subprocess.PIPE)
    result.wait()
    return_code = result.returncode
    out, err = result.communicate()

    value = None
    if return_code == 0:
        for line in out.splitlines():
            if not line.startswith('>>>'):
                continue
            value = line.split(':',1)[1].lstrip()
            print
    else:
        print 'non-zero exit', return_code
    print 'return value:', value

if __name__ == '__main__':
    func()

Then run python lvl0.py to check the output to be

argv ['a', 'b']

return value: time

Now bring these under your revision control system and start making changes a few lines at a time, every time running python lvl0.py to check what you might have broken. Commit each revision so you can roll back to last "known good" state and slowly bring in the rest of your code.