Unexpected behavior of mypy with TypeVar, Generic, and decorators

Question

I'm currently in trouble to understand TypeVar and Generic in Python.

Here's the setting of my MWE (Minimal Working Example) :

1. I defined an abstract class, which has a __call__ method:
   • This method performs some complicated computation.
   • Its argument type is ValueT, a generic type that will be specified when a concrete subclass is implemented.
2. The abstract class also defines a method that converts an argument of type AcceptedValueT into type ValueT.
   • To avoid repeating the same code (since I’ll also have some pre/post processing to do), I wrap this with a decorator.
   • This also lets me follow the Open/Closed Principle (OCP) by keeping the conversion logic abstract.

My questions are:

• Why does mypy accept calls like complicated_function(3) and complicated_function("3") when I’ve set AcceptedValueT to float ?
• Why does new_value inside the decorator have the type Any according to reveal_type?

from abc import abstractmethod
from dataclasses import dataclass
from typing import Callable, Union, TypeVar, Generic, reveal_type

ValueT = TypeVar("ValueT")
AcceptedValueT = TypeVar("AcceptedValueT")
SelfT = TypeVar("SelfT", bound="AbstractFancyClass")


def decorator(
        func: Callable[[SelfT, ValueT], ValueT],
) -> Callable[[SelfT, AcceptedValueT], ValueT]:
    print("Start decorator")

    def wrapper(self: SelfT, value: AcceptedValueT) -> ValueT:
        print("Start wrapper")
        new_value = self.convert_to_value_type(value)
        reveal_type(new_value)  # Why new_value has the Any type ?
        print("End wrapper")
        return func(self, new_value)

    print("End decorator")
    return wrapper


@dataclass
class AbstractFancyClass(Generic[ValueT, AcceptedValueT]):
    value_: ValueT

    @abstractmethod
    def __call__(self, value: ValueT) -> ValueT:
        raise NotImplementedError

    @abstractmethod
    def convert_to_value_type(self, strange_value: AcceptedValueT) -> ValueT:
        raise NotImplementedError


@dataclass
class FancyClass(AbstractFancyClass[float, float]):
    value_: float

    @decorator
    def __call__(self, value: float) -> float:
        print("Start __call__")
        value = value ** 2 + 2.0
        print("End __call__")
        return value

    def convert_to_value_type(self, strange_value: float) -> float:
        return strange_value


complicated_function = FancyClass(1.0)
print("\nWe use an instance of the FancyClass\n")
a = complicated_function(3.0)
print(f"a = {a}")
b = complicated_function(3)  # Why mypy says it's ok ?
print(f"b = {b}")  
c = complicated_function("3")  # Why mypy says it's ok ?
print(f"c = {c}")

assert a == b == c == 11.0

Answer 1

The Python typing system is only first order, and does not allow for Higher Kinded Types (HKTs). What this means in practice is that you cannot abstract over generic types (like you do when you write SelfT = TypeVar("SelfT", bound="AbstractFancyClass")). That is, AbstractFancyClass is generic, and SelfT is abstracting over all possible type parameters that AbstractFancyClass can be parameterised with with. For example, you want to be able to say things like SelfT[str, str] and have that translate to AbstractFancyClass[str, str]. You cannot do that with Python's typing system at present.

Alternatives

You are indirectly using the Template Method Pattern. If you switch to a more standard way of implementing such a pattern, a static type checker will be able to recognise incorrect uses of your class. When using the Template Method Pattern you provide a concrete function in an abstract class that makes use of one or more abstract functions. It is up to child classes to fill in these implementations. This in effect what you are already doing, we're just shifting where the concrete implementation is defined (from the decorator to the class) to help the static type checker.

With your example code, we would provide __call__(AcceptedValueT) -> ValueT. This function would use an abstract convert() function and an abstract _call_impl() function. The former to do the conversion from AcceptedValueT to ValueT, and the latter to perform the actual logic for the __call__ call.

Your example code would now look like:

from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import Generic, TypeVar

ValueT = TypeVar("ValueT")
AcceptedValueT = TypeVar("AcceptedValueT")

@dataclass
class AbstractFancyClass(Generic[AcceptedValueT, ValueT], ABC):
    value_: ValueT

    def __call__(self, value: AcceptedValueT) -> ValueT:
        # concrete implementation that delegates to two template (abstract)
        # functions
        return self._call_impl(self.convert(value))

    @abstractmethod
    def _call_impl(self, value: ValueT) -> ValueT:
        # will do the actual work of `__call__`
        raise NotImplementedError

    @abstractmethod
    def convert(self, accepted: AcceptedValueT) -> ValueT:
        raise NotImplementedError

@dataclass
class FancyClass(AbstractFancyClass[str, float]):
    value_: float

    def convert(self, accepted: str) -> float:
        return float(accepted)

    def _call_impl(self, value: float) -> float:
        # simple pass-through function
        print("Start __call__")
        ...
        print("End __call__")
        return value

complicated_function = FancyClass(1.0)
print()
print("We use an instance of the FancyClass")
a = complicated_function("3")
print(f"a = {a}")  # prints: a = 3.0

b = complicated_function(3)  # ERROR
# error: Argument 1 to "__call__" of "AbstractFancyClass" has
#       incompatible type "int"; expected "str"  [arg-type]

c = complicated_function(3.0)  # ERROR
# error: Argument 1 to "__call__" of "AbstractFancyClass" has
#     incompatible type "float"; expected "str"  [arg-type]