Supervisor events: how to listen for PROCESS_STATE_STOPPED events for all or specified processes

Question

I'm trying to use supervisord events to listen to PROCESS_STATE_STOPPED events from processes managed by Supervisor.

My event listener (listener.py), looks like this:

import sys

from supervisor.childutils import listener

def write_stdout(s):
    sys.stdout.write(s)
    sys.stdout.flush()


def write_stderr(s):
    sys.stderr.write(s)
    sys.stderr.flush()


def main():
    while True:
        headers, body = listener.wait(sys.stdin, sys.stdout)
        body = dict([pair.split(":") for pair in body.split(" ")])
        write_stderr("Headers: %r\n" % repr(headers))
        write_stderr("Body: %r\n" % repr(body))
        listener.ok(sys.stdout)

        if headers["eventname"] == "PROCESS_STATE_STOPPED":
            write_stderr("Process state stopped...\n")


if __name__ == '__main__':
    main()

My corresponding entry in supervisord.conf is as follows:

[program:theprogramname]
command=/bin/cat              ; the program (relative uses PATH, can take args)
process_name=%(program_name)s ; process_name expr (default %(program_name)s)
numprocs=1                    ; number of processes copies to start (def 1)
...
[eventlistener:theeventlistenername]
command=python /home/mickm/listener.py    ; the program (relative uses PATH, can take args)
process_name=%(program_name)s_%(process_num)s       ; process_name expr (default %(program_name)s)
numprocs=1                           ; number of processes copies to start (def 1)
events=PROCESS_STATE                 ; event notif. types to subscribe to (req'd)
autorestart=true
redirect_stderr=true

I've researched the other StackOverflow questions on this area & I've attempted to implement accepted solutions from:

• How to subscribe to PROCESS_STATE_RUNNING events for all processes
• Supervisor event subscription is hanging on READY state

However, when I run listener.py , it outputs READY to SDOUT & goes no further. I've tried stopping/starting & restarting supervisor-monitored processes, but my script does pick up anything.

To test this, I'm:

1. running my event listener script in terminal A;
2. performing a kill -9 <pid> on the /bin/cat process in terminal B.

My supervisor log is as follows:

2017-01-13 14:56:36,168 INFO success: theeventlistenername_0 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-01-13 14:56:36,168 INFO success: theprogramname entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-01-13 14:57:29,457 INFO exited: theprogramname (terminated by SIGKILL; not expected)
2017-01-13 14:57:30,460 INFO spawned: 'theprogramname' with pid 25788
2017-01-13 14:57:31,462 INFO success: theprogramname entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

While the the supervisor log knows my process has been terminated:

i) Nothing is detected by listener.py, which remains at 'READY' prompt.

ii) supervisor restarts to process (as expected).

I'm wondering why the listener is not picking up the PROCESS_STATE_STOPPED event for the program defined in the config file (as per Supervisor event docs). Also - whether this can be applied generally for all supervisor-managed processes.

Thanks.

Answer 1

Configuration variable redirect_stderr:

http://supervisord.org/configuration.html

Do not set redirect_stderr=true in an [eventlistener:x] section. Eventlisteners use stdout and stdin to communicate with supervisord. If stderr is redirected, output from stderr will interfere with the eventlistener protocol.

This is my working script:

import sys

from supervisor.childutils import listener

def write_stdout(s):
    sys.stdout.write(s)
    sys.stdout.flush()

def write_stderr(s):
    sys.stderr.write(s)
    sys.stderr.flush()

def main():
    while True:
        headers, body = listener.wait(sys.stdin, sys.stdout)
        body = dict([pair.split(":") for pair in body.split(" ")])

        write_stderr("Headers: %r\n" % repr(headers))
        write_stderr("Body: %r\n" % repr(body))

        if headers["eventname"] == "PROCESS_STATE_STOPPING":
            write_stderr("Process state stopping...\n")

        # acknowledge the event
        write_stdout("RESULT 2\nOK")

But I believe you cannot test it simply running your script. Your supervisorctl .conf file has some special env variables like events which doesnt appear when you run your script manually

If you want to test it then add some logging to your listener:

[eventlistener:theeventlistenername]
command=python /home/mickm/listener.py    ; the program (relative uses PATH, can take args)
process_name=%(program_name)s_%(process_num)s       ; process_name expr (default %(program_name)s)
numprocs=1                           ; number of processes copies to start (def 1)
events=PROCESS_STATE           ; event notif. types to subscribe to (req'd)
autorestart=true
stderr_logfile=errorlogfile
stdout_logfile=applogfile

Also you can debug headers variable and you can check the data you are receiving.

With the logging enabled you can test it with a tail -f error.log.path on the stderr_logfile logfile.

 Headers: "{'ver': '3.0', 'poolserial': '4', 'len': '71', 'server': 'supervisor', 'eventname': 'PROCESS_STATE_RUNNING', 'serial': '4', 'pool': 'mylistener'}" 
 Body: "{'from_state': 'STARTING', 'processname': 'someprocess', 'pid': '345', 'groupname': 'someprocess'}" 
 Process state running...

And when I kil -9 the process the above output appears in the log.

Also the stdout_logfile will log this kind of information:

RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY