Can't override __init__ of class from Cython extension

Question

I am trying to subclass pysam's Tabixfile class and add additional attributes on instantiation.

class MyTabixfile(pysam.Tabixfile):

    def __init__(self, filename, mode='r', *args, **kwargs):
        super().__init__(filename, mode=mode, *args, **kwargs)
        self.x = 'foo'

When I try to instantiate my MyTabixfile subclass, I get a TypeError: object.__init__() takes no parameters:

>>> mt = MyTabixfile('actn2-oligos-forward.tsv.gz')
Traceback (most recent call last):
  File "<ipython-input-11-553015ac7d43>", line 1, in <module>
    mt = MyTabixfile('actn2-oligos-forward.tsv.gz')
  File "mytabix.py", line 4, in __init__
    super().__init__(filename, mode=mode, *args, **kwargs)
TypeError: object.__init__() takes no parameters

I also tried calling the Tabixfile constructor explicitly:

class MyTabixfile(pysam.Tabixfile):

    def __init__(self, filename, mode='r', *args, **kwargs):
        pysam.Tabixfile.__init__(self, filename, mode=mode, *args, **kwargs)
        self.x = 'foo'

but this still raises TypeError: object.__init__() takes no parameters.

This class is actually implemented in Cython; the constructor code is below:

cdef class Tabixfile:
    '''*(filename, mode='r')*

    opens a :term:`tabix file` for reading. A missing
    index (*filename* + ".tbi") will raise an exception.
    '''
    def __cinit__(self, filename, mode = 'r', *args, **kwargs ):
        self.tabixfile = NULL
        self._open( filename, mode, *args, **kwargs )

I read through the Cython documentation on __cinit__ and __init__ which says

Any arguments passed to the constructor will be passed to both the __cinit__() method and the __init__() method. If you anticipate subclassing your extension type in Python, you may find it useful to give the __cinit__() method * and ** arguments so that it can accept and ignore extra arguments. Otherwise, any Python subclass which has an __init__() with a different signature will have to override __new__() 1 as well as __init__(), which the writer of a Python class wouldn’t expect to have to do.

The pysam developers did take the care to add *args and **kwargs to the Tabixfile.__cinit__ method, and my subclass __init__ matches the signature of __cinit__ so I do not understand why I'm unable to override the initialization of Tabixfile.

I'm developing with Python 3.3.1, Cython v.0.19.1, and pysam v.0.7.5.

Answer 1

The documentation is a little confusing here, in that it assumes that you're familiar with using __new__ and __init__.

The __cinit__ method is roughly equivalent to a __new__ method in Python.*

Like __new__, __cinit__ is not called by your super().__init__; it's called before Python even gets to your subclass's __init__ method. The reason __cinit__ needs to handle the signature of your subclass __init__ methods is the exact same reason __new__ does.

If your subclass does explicitly call super().__init__, that looks for an __init__ method in a superclass—again, like __new__, a __cinit__ is not an __init__. So, unless you've also defined an __init__, it will pass through to object.

---

You can see the sequence with the following code.

cinit.pyx:

cdef class Foo:
    def __cinit__(self, a, b, *args, **kw):
        print('Foo.cinit', a, b, args, kw)
    def __init__(self, *args, **kw):
        print('Foo.init', args, kw)

init.py:

import pyximport; pyximport.install()
import cinit

class Bar(cinit.Foo):
    def __new__(cls, *args, **kw):
        print('Bar.new', args, kw)
        return super().__new__(cls, *args, **kw)
    def __init__(self, a, b, c, d):
        print('Bar.init', a, b, c, d)
        super().__init__(a, b, c, d)

b = Bar(1, 2, 3, 4)

When run, you'll see something like:

Bar.new (1, 2, 3, 4) {}
Foo.cinit 1 2 (3, 4) {}
Bar.init 1 2 3 4
Foo.init (1, 2, 3, 4) {}

---

So, the right fix here depends on what you're trying to do, but it's one of these:

1. Add an __init__ method to the Cython base class.
2. Remove the super().__init__ call entirely.
3. Change the super().__init__ to not pass any params.
4. Add an appropriate __new__ method to the Python subclass.

I suspect in this case it's #2 you want.

---

* It's worth noting that __cinit__ definitely isn't identical to __new__. Instead of getting a cls parameter, you get a partially-constructed self object (where you can trust __class__ and C attributes but not Python attributes or methods), the __new__ methods of all classes in the MRO have already been called before any __cinit__; the __cinit__ of your bases gets called automatically instead of manually; you don't get to return a different object besides the one that's been requested; etc. It's just that it's called before the __init__, and expected to take pass-through parameters, in the same way as __new__ is.

Answer 2

I would have commented rather than posting an answer but I don't have enough StackOverflow foo as yet.

@abarnert's post is excellent and very helpful. I would just add a few pysam specifics here as I have just done subclassing on pysam.AlignmentFile in a very similar way.

Option #4 was the cleanest/easiest choice which meant only changes in my own subclass __new__ to filter out the unknown params:

def __new__(cls, file_path, mode, label=None, identifier=None, *args, **kwargs):
    # Suck up label and identifier unknown to pysam.AlignmentFile.__cinit__
    return super().__new__(cls, file_path, mode, *args, **kwargs)

It should also be noted that the pysam file classes don't seem to have explicit __init__ method's, so you also need to omit param pass through as that goes straight to object.__init__ which does not accept parameters:

def __init__(self, label=None, identifier=None, *args, **kwargs):
    # Handle subclass params/attrs here
    # pysam.AlignmentFile doesn't have an __init__ so passes straight through to
    # object which doesn't take params. __cinit__ via new takes care of params
    super(pysam.AlignmentFile, self).__init__()