What is the meaning of __total__ dunder attribute in Python 3?

Question

In newly released Python 3.8 there is a new type annotation typing.TypedDict. Its documentation mentions that

The type info for introspection can be accessed via Point2D.__annotations__ and Point2D.__total__. [....]

While __annotations__ is well-known, having been introduced in PEP 3107, I cannot find any information on __total__. Could anyone explain its meaning and if possible linking to authoritative sources?

Answer 1

I am guessing that the __total__ field signifies whether instances must be complete (the default) or not (all fields optional). I started my search at PEP 589, which introduced TypedDict and describes totality as such. It used a total argument, which it would make sense to rename dunder-style for the class syntax. However, I did not find when such a renaming took place.

Looking into MyPy, which is the actual type checker that cares about these annotations, there is similar documentation on TypedDict and totality, but again no reference to the dunder syntax. Digging into its implementation led to more confusion, as TypedDictType in types.py doesn't have a total field, but separate items and required_keys. Totality would imply that items.keys()==required_keys but the implementation makes different assumptions, such as can_be_false relying on items alone. total=False should in principle mean required_keys is empty.

The CPython source for _TypedDictMeta at least reveals that the total argument and __total__ dunder are one and the same, although the source describes TypedDict itself as "may be added soon".

Answer 2

TypedDict was accepted in Python 3.8 via PEP 589. From Python, it appears __total__ is a boolean flag set to True by default:

tot = TypedDict.__total__
print(type(tot))
print(tot)

# <class 'bool'>
# True

As mentioned in other posts, details on this method are limited in the docs, but @Yann Vernier's link to the CPython source code strongly suggests __total__ is related to the new total keyword introduced in Python 3.8:

# cypthon/typing.py

class _TypedDictMeta(type):
    def __new__(cls, name, bases, ns, total=True):
        """Create new typed dict class object.
        ...
        """
        ...
        if not hasattr(tp_dict, '__total__'):
            tp_dict.__total__ = total
        ...

How does it work?

Synopsis: by default, all keys are required when instantiating a defined TypedDict. total=False overrides this restriction and allows optional keys. See the following demonstration.

Given

A test directory tree:

Code

Files in the test directory:

# rgb_bad.py

from typing import TypedDict


class Color(TypedDict):
    r: int
    g: int
    b: int
    a: float


blue = Color(r=0, g=0, b=255)                     # missing "a"

# rgb_good.py

from typing import TypedDict


class Color(TypedDict, total=False):
    r: int
    g: int
    b: int
    a: float


blue = Color(r=0, g=0, b=255)                     # missing "a"

Demo

If a key is missing, mypy will complain at the commandline:

> mypy code/rgb_bad.py
code\rgb_bad.py:11: error: Key 'a' missing for TypedDict "Color"
...

Setting total=False permits optional keys:

> mypy code/rgb_good.py
Success: no issues found in 1 source file

See Also

• Tweet by R. Hettinger demonstating totality
• PEP section on totality in PEP 589
• Article Section on types and TypedDict in Python 3.8 by Real Python
• typing-extensions package to use TypedDict in Python 3.5, 3.6