Base class deleted before subclass during python __del__ processing

Question

Context

I am aware that if I ask a question about Python destructors, the standard argument will be to use contexts instead. Let me start by explaining why I am not doing that.

I am writing a subclass to logging.Handler. When an instance is closed, it posts a sentinel value to a Queue.Queue. If it doesn't, a second thread will be left running forever, waiting for Queue.Queue.get() to complete.

I am writing this with other developers in mind, so I don't want a failure to call close() on a handler object to cause the program to hang.

Therefore, I am adding a check in __del__() to ensure the object was closed properly.

I understand circular references may cause it to fail in some circumstances. There's not a lot I can do about that.

Problem

Here is some simple example code:

explicit_delete = True

class Base:
    def __del__(self):
        print "Base class cleaning up."

class Sub(Base):
    def __del__(self):
        print "Sub-class cleaning up."
        Base.__del__(self)

x = Sub()

if explicit_delete:
    del x

print "End of thread"

When I run this I get, as expected:

Sub-class cleaning up.
Base class cleaning up.
End of thread

If I set explicit_delete to False in the first line, I get:

End of thread
Sub-class cleaning up.
Exception AttributeError: "'NoneType' object has no attribute '__del__'" in <bound method Sub.__del__ of <__main__.Sub instance at 0x00F0B698>> ignored

It seems the definition of Base is removed before the x.__del__() is called.

The Python Documentation on __del__() warns that the subclass needs to call the base-class to get a clean deletion, but here that appears to be impossible.

Can you see where I made a bad step?

Answer 1

Your code is slightly misleading, I tried it and it failed just as you described. But then I wrote something like this:

import threading

class Base( object ):
    def __del__(self):
        print "Base class cleaning up."

class Sub(Base):
    def __del__(self):
        print "Sub-class cleaning up."
        Base.__del__( self )

def f():
    x = Sub()
    print "End of thread"

t = threading.Thread( target = f )
t.start()
t.join()

and the output was:

End of thread
Sub-class cleaning up.
Base class cleaning up.
End of main thread.

So I guess you cannot rely on __del__ methods during interpreter shutdown (I think that class objects are collected before instances?), but before that point they work just as expected.

Maybe keeping main thread alive until others are dead and not creating your Handler subclass instances in main thread would be enough?