TypeError: can't pickle file objects webpy and subprocess

Question

I am trying to make this following code possible. But I keep getting "TypeError: can't pickle file objects" error. I am both new to python and webpy. Can anyone tell how can I achieve this code.

import web 
import subprocess

web.config.debug = False
urls = ( 
   "/start", "start",
   "/end", "end"
)
app = web.application(urls, locals())
s = web.session.Session(app, web.session.DiskStore('sessions'), initializer={"p": None})

class start:
    def GET(self):
        s.p = subprocess.Popen(['ls', '-a'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        return ""

class end:
    def GET(self):
        out, err = s.p.communicate();
        return out 

if __name__ == "__main__":
    app.run()

the error I keep seeing is python27/lib/python2.7/copy_reg.py", line 70, in _reduce_ex raise TypeError, "can't pickle %s objects" % base.name TypeError: can't pickle file objects

Answer 1

I am, myself, a webpy newbie. Nevertheless, after doing some "research", it seems that webpy cannot pickle the subprocess.Popen object[1]. So, lets try the following approach -- that is, creating it in the end response and printing its output.

In other words:

import web 
import subprocess

web.config.debug = False
urls = ( 
   "/start", "start",
   "/end", "end"
)
app = web.application(urls, locals())
s = web.session.Session(
        app, web.session.DiskStore('sessions'), initializer={"p": None})

class start:
    def GET(self):
        s.p = ['ls', '-a']
        return ""

class end:
    def GET(self):
        out, err = subprocess.Popen(
                       s.p , stdout=subprocess.PIPE,
                       stderr=subprocess.PIPE).communicate()
        s.kill() # kill the session.
        return out 

if __name__ == "__main__":
    app.run()

[1] As you can see:

import pickle
import subprocess

with open('bla', 'wb') as f:
    pickle.dump(subprocess.Popen(['ls'],
                stdout=subprocess.PIPE,
                stderr=subprocess.PIPE), f)

The subprocess.Popen returns a file descriptor and from the traceback I infer that pickle cannot serialize file descriptors.

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib64/python2.7/pickle.py", line 1370, in dump
    Pickler(file, protocol).dump(obj)
  File "/usr/lib64/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib64/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib64/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib64/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib64/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib64/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib64/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/usr/lib64/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle file objects

In addition to that, the Pickle documentation mentions that it supports only booleans, integers, floating point numbers, complex numbers, strings, bytes objects, byte arrays, and None. Thus, it seems like hopeless.

EDIT

If you insist on creating the process' file-descriptor, when receiving a start request from the client, You can use a dictionary to map session to a file-descriptor.

It should be like this:

s = web.session.Session(
        app, web.session.DiskStore('sessions'), initializer={"p": None})
session_map = {}

class start:
    def GET(self):
        session_map[s] = subprocess.Popen(['ls', '-a'],
                                          stdout=subprocess.PIPE,
                                          stderr=subprocess.PIPE)
        return ""

class end:
    def GET(self):
        out, err = session_process[s].communicate()
        session_process.pop(s, None)
        s.kill
        return out