Python custom JSON encoder isn't handling numpy NaN or Infinite values

Question

So I'm trying to write a custom encoder that handles different numpy values:

class NumpyEncoder(DjangoJSONEncoder):
    def default(self, obj):
        if isinstance(obj, np.integer):
            return int(obj)
        elif isinstance(obj, np.floating):
            return float(obj)
        elif isinstance(obj, np.ndarray):
            return obj.tolist()
        if np.isnan(obj) or np.isinf(obj):
            return None
        else:
            return super().default(obj)

However, it doesn't appear to be handling np.NaN or np.inf objects:

obj = {'test': np.NaN, 'test2': np.inf}
json.dumps(obj, cls=LazyNumpyEncoder)
Out[5]: '{"test": NaN, "test2": Infinity}'

I've also tried using elif isinstance(obj, np.nan):, but that hasn't worked either.

What I want is my NaN and Infinite (including negative infinite values) to be turned into nulls.

Anyone know where I've gone wrong?

Answer 1

So @Vovanrock explained it super well, but I'll post what I did to fix the problem. Before running the dict through dumps I simply converted the inf and nan objects to strings:

def _convert_numpy_objects(self, dict_to_convert : Dict) -> Dict:
    new = {}
    for k, v in dict_to_convert.items():
        if isinstance(v, dict):
            new[k] = self._convert_numpy_objects(v)
        else:
            if isinstance(v, float) and (np.isnan(v) or np.isinf(v)):
                new[k] = str(v)
            else:
                new[k] = v
    return new

Answer 2

The reason here is that default method can be used only to make new types serializable. It can't be used to override serializing of types, already defined in json. Here's what the docstring of JSONEncoder says (emphasis mine):

To extend this to recognize other objects, subclass and implement a .default() method with another method that returns a serializable object for o if possible, otherwise it should call the superclass implementation (to raise TypeError).

If you look at the sources, you'll notice that default is called after all other serialization attempts fail. So the code in OP post doesn't get called. In accordance with the above citation, DjangoJSONEncoder implements default to make it understand date/time, decimal types, and UUIDs.

As for how to work around this - well, I wouldn't bother with subclassing but rather go down the dict recursively (beware loops!) and replace NaNs and Infs with None before passing the dict into json.dumps. It is also possible to use json.dumps(..., allow_nan=False) to make sure all NaNs are caught before invoking dumps.

Another option - subclass JSONEncoder / DjangoJSONEncoder, override dumps, insert the algorithm mentioned above and pass rest of the work to a parent class.