How to write a typehint / use a typevar for a filter method?

Question

I have a helper method in Python that returns a list of methods and annotated data for each method. So it's a dict of lists. Annotated data is expressed by an Attribute class.

The definition is as follows:

# A filter predicate can be either an attribute object or a tuple/list of attribute objects.
AttributeFilter = Union['Attribute', Iterable['Attribute'], None]

# A class offering a helper method
class Mixin:
  def GetMethods(self, filter: AttributeFilter=Attribute) -> Dict[Callable, List[Attribute]]:
    pass

This syntax and the corresponding type check works fine.
Of cause I would like to improve it.

---

Users normally derive user-defined attributes from class Attribute. I would like to express, if a user passes a derived class like UserAttribute to GetMethods, that is returns a dict of lists of UserAttributes.

# Some user-defined attribute and some public data in it
class UserAttribute(Attribute):
  someData: str

# Create a big class
class Big(mixin):
  
  # Annotate a method with meta information
  @UserAttribute("hello")
  def method(self):
    pass

# Create an instance
prog = Big()

# search for all methods that have 'UserAttribute' annotations
methods = prog.GetMethods(filter=UserAttribute)
for method, attributes in methods:
  for attribute in attributes:
    print(attribute.someData)

This code can be executed without problems, but PyCharm's type checker doesn't know that field someData exists for attribute in the last line (print call).

Possible solution 1: I could use a typehint for every variable getting a return value from GetMethods like this:

methods:Dict[Callable, List[UserAttribute]] = prog.GetMethods(filter=UserAttribute)

This approach replicates a lot of code.

Possible solution 2: Is it possible to abstract Dict[Callable, List[UserAttribute]] into some kind of new generic so I could use:

# pseudo code
UserGeneric[G] := Dict[Callable, List[G]]

# shorter usage
methods:UserGeneric[UserAttribute] = prog.GetMethods(filter=UserAttribute)

Possible solution 3: At best I would like to use a TypeVar like this:

Attr = TypeVar("Attr", Attribute)

# A filter predicate can be either an attribute object or a tuple/list of attribute objects.
AttributeFilter = Union[Attr, Iterable[Attr], None]

# A class offering a helper method
class Mixin:
  def GetMethods(self, filter: AttributeFilter=Attribute) -> Dict[Callable, List[Attr]]:
    pass

Unfortunately, TypeVar expects at least 2 constraints like T = TypeVar("T", str, byte).

---

At the end, this is a more complex variant of the simple example shown in the typing manual pages:

T = TypeVar("T")

def getElement(l: List[T]) -> T:
  pass

Final question:
How to constrain a TypeVar T to certain objects of a class and all it's subclasses, without the need of a union like in the str vs. byte example above.

This question is related to https://github.com/Paebbels/pyAttributes.

Answer 1

Unfortunately, TypeVar expects at least 2 constraints like T = TypeVar("T", str, byte).

Actually, TypeVar can work with just one constraint. To do so, you would do something like T = TypeVar("T", bound=str).

For more details, I would recommend reading the mypy docs on TypeVars with upper bounds -- the official typing docs are regrettably not very polished and often cover important concepts like TypeVars with upper bounds very briefly.

---

So, that means you can solve your problem by doing this:

from typing import TypeVar, Union, Iterable, Dict, Callable, List

class Attribute: pass

class UserAttribute(Attribute): pass


TAttr = TypeVar("TAttr", bound=Attribute)


AttributeFilter = Union[TAttr, Iterable[TAttr], None]

class Mixin:
  def GetMethods(self,
                 filter: AttributeFilter[TAttr] = Attribute,
                 ) -> Dict[Callable, List[TAttr]]:
    pass

m = Mixin()

# Revealed type is 'builtins.dict[def (*Any, **Any) -> Any, builtins.list[test.UserAttribute*]]'
reveal_type(m.GetMethods([UserAttribute(), UserAttribute()]))

Some notes:

1. I named my TypeVar TAttr, not Attr. You want to make it obvious to the reader what the "placeholders" in your signatures are, so semi-common convention is to prefix your TypeVars with either T or _T. (And if you don't need an upper-bound and instead want a truly open-ended placeholder, the convention is to use a single uppercase letter like T or S.)

2. In GetMethods(...), you need to do AttributeFilter[TAttr]. If you do just AttributeFilter, that's actually equivalent to doing AttributeFilter[Any]. This is for the same reason why doing List means the same thing as List[Any].

3. I'm reusing TAttr to both define the type alias and GetMethods mostly for convenience, but you could also create a new TypeVar and use that for GetMethods if you really want: that would mean the exact same thing.

4. reveal_type(...) is a special pseudo-function that some type checkers (e.g. mypy and pyre) understand: it makes the type checker print out what it thinks the type of the expression is.

5. Not all type checkers (such as mypy) may support setting a default argument for a generic type. If your type checker complains, you can probably work around it by creating an overload with a no-args and single-arg variants.