How to incorporate data from two distinct sources (that don't have a RDBMS relationship) in a single serializer?

Question

I'm trying to serialize some objects whose data is stored in 2 databases, linked by common UUIDs. The second database DB2 stores personal data, so it is run as a segregated microservice to comply with various privacy laws. I receive the data as a decoded list of dicts (rather than an actual queryset of model instances). How can I adapt the ModelSerializer to serialize this data?

Here's a minimal example of interacting with DB2 to get the personal data:

# returns a list of dict objects, approx representing PersonalData.__dict__
# `custom_filter` is a wrapper for the Microservice API using `requests`
personal_data = Microservice.objects.custom_filter(uuid__in=uuids)

And here's a minimal way of serializing it, including the date of birth:

class PersonalDataSerializer(serializers.Serializer):
    uuid = serializers.UUIDField() # common UUID in DB1 and DB2
    dob = serializers.DateField() # personal, so can't be stored in DB1

In my application, I need to serialize the Person queryset, and related personal_data, into one JSON array.

class PersonSerializer(serializers.ModelSerializer):
    dob = serializers.SerializerMethodField()
    # can't use RelatedField for `dob` because the relationship isn't
    # codified in the RDBMS, due to it being a separate Microservice.

    class Meta:
        model = Person
        # A Person object has `uuid` and `date_joined` fields.
        # The `dob` comes from the personal_data, fetched from the Microservice
        fields = ('uuid', 'date_joined', 'dob',)

    def get_dob(self):
        raise NotImplementedError # for the moment

I don't know if there's a nice DRF way to link the two. I definitely don't want to be sending (potentially thousands of) individual requests to the microservice by including a single request in get_dob. The actual view just looks like this:

class PersonList(generics.ListAPIView):
    model = Person
    serializer_class = PersonSerializer

    def get_queryset(self):
        self.kwargs.get('some_filter_criteria')
        return Person.objects.filter(some_filter_criteria)

Where should the logic go to link the microservice data into the serializer, and what should it look like?

Answer 1

I suggest you to override the serializer and your list method.

Serializer:

class PersonSerializer(models.Serializer):
    personal_data = serializers.DictField()

    class Meta:
        model = Person

make a function to add personal_data dictionary to persons object. Use this method before giving the list of person objects to the serializer.

def prepare_persons(persons):
    person_ids = [p.uuid for p in persons]
    personal_data_list = Microservice.objects.custom_filter(uuid__in=person_ids)
    personal_data_dict = {pd['uuid']: pd for pd in personal_data_list}
    for p in persons:
        p.personal_data = personal_data_dict[p.id]
    return persons


def list(self, request, *args, **kwargs):

    queryset = self.filter_queryset(self.get_queryset())

    page = self.paginate_queryset(queryset)

    if page is not None:
        page = prepare_persons(page)
        serializer = self.get_serializer(page, many=True)
        return self.get_paginated_response(serializer.data)
    else:
        persons = prepare_persons(queryset)

    serializer = self.get_serializer(persons, many=True)
    return Response(serializer.data)

Answer 2

Because you want to only hit your database one time, a good way to add your extra data to your queryset is by adding a custom version of ListModelMixin to your ViewSet that includes extra context:

class PersonList(generics.ListAPIView):
    ...

    def list(self, request, *args, **kwargs):
        queryset = self.filter_queryset(self.get_queryset())
        # Pseudo-code for filtering, adjust to work for your use case
        filter_criteria = self.kwargs.get('some_filter_criteria')
        personal_data = Microservice.objects.custom_filter(filter_criteria)

        page = self.paginate_queryset(queryset)
        if page is not None:
            serializer = self.get_serializer(
                page, 
                many=True, 
                context={'personal_data': personal_data}
            )
            return self.get_paginated_response(serializer.data)

        serializer = self.get_serializer(
            queryset, 
            many=True, 
            context={'personal_data': personal_data}
        )
        return Response(serializer.data)

Then, access the extra context in your serializer by overriding the to_representation method:

def to_representation(self, instance):
    """Add `personal_data` to the object from the Microservice"""
    ret = super().to_representation(instance)
    personal_data = self.context['personal_data']
    ret['personal_data'] = personal_data[instance.uuid]
    return ret