Systemd + non-root Gunicorn service = defunct subprocess

Question

I'm following this document to setup a Systemd socket and service for my gunicorn server.

• Systemd starts gunicorn as www-data
• gunicorn forks itself (default behavior)
• the server starts a subprocess with subprocess.Popen()
• the subprocess finishes without an error, but the parent keeps getting None from from p.poll() instead of an exit code
• the subprocess ends up defunct

Here's the process hierarchy:

$ ps eauxf
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
...
www-data 14170  0.0  0.2  65772 20452 ?        Ss   10:57   0:00 /usr/bin/python /usr/bin/gunicorn digits.webapp:app --pid /run/digits/pid --config /usr/lib/python2.7/dist-packages/digits/gunicorn_config.py
www-data 14176  0.8  3.4 39592776 283124 ?     Sl   10:57   0:05  \_ /usr/bin/python /usr/bin/gunicorn digits.webapp:app --pid /run/digits/pid --config /usr/lib/python2.7/dist-packages/digits/gunicorn_config.py
www-data 14346  5.0  0.0      0     0 ?        Z    11:07   0:01      \_ [python] <defunct>

Here's the kicker: when I run the service as root instead of www-data, everything works as expected. The subprocess finishes and the parent immediately gets the child's return code.

/lib/systemd/system/digits.service

[Unit]
Description=DIGITS daemon
Requires=digits.socket
After=local-fs.target network.target
[Service]
PIDFile=/run/digits/pid
User=www-data
Group=www-data
Environment="DIGITS_JOBS_DIR=/var/lib/digits/jobs"
Environment="DIGITS_LOGFILE_FILENAME=/var/log/digits/digits.log"
ExecStart=/usr/bin/gunicorn digits.webapp:app \
    --pid /run/digits/pid \
    --config /usr/lib/python2.7/dist-packages/digits/gunicorn_config.py
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
PrivateTmp=true
[Install]
WantedBy=multi-user.target

/lib/systemd/system/digits.socket

[Unit]
Description=DIGITS socket
[Socket]
ListenStream=/run/digits/socket
ListenStream=0.0.0.0:34448
[Install]
WantedBy=sockets.target

/usr/lib/tmpfiles.d/digits.conf

d /run/digits 0755 www-data www-data -

Answer 1

I ran into the same issue today on CentOS-7. I finally overcame it by ignoring the instructions in this document -- which indicates to use the /run/ hierarchy within which to create the socket -- and I instead used /tmp/. That worked.

Note that my PID file is still placed underneath /run/ (no issues there).

In summary, instead of placing your socket somewhere underneath /run/..., try placing it somewhere underneath /tmp/... instead. It worked for me on CentOS-7 with systemd.