Override model serializer globally, big integer field

[HoodyH]

First of all thanks for this project, i can't use Django without drf 😀

Im looking for a way to override serializers.ModelSerializer globally

Little bit of context:
Our application is grown big and we want to switch django backend from BigAutoField to SnowflakeAutoField.

My problem:

• ts/js (My frontend) do not support numbers over 2^52,
• I don't want to use Bigint in ts
• i don't want to touch MAX_SAFE_INTEGER

So the best solution in my opinion is to serialize bigint as string.

class SnowflakeField(serializers.Field):

    def to_representation(self, value):
        return str(value)

    def to_internal_value(self, data):
        try:
            return int(data)
        except ValueError:
            raise serializers.ValidationError("Invalid integer value")

class SnowflakeModelSerializer(serializers.ModelSerializer):
    """
    Snowflake is a bigint so convert to string to avoid frontend memory handling problems
    """
    def build_standard_field(self, field_name, model_field):

        field_class, field_kwargs = super().build_standard_field(field_name, model_field)

        # Use BigIntAsStringField per BigIntegerField
        if isinstance(model_field, models.BigIntegerField):
            field_class = SnowflakeField

        return field_class, field_kwargs

Anyway, since django uses bigint as id, serializing it as sting could be a better option in any case since the id can grow big

Possible ways

1. use my custom serializer around the code (not so clean)

class MyModelSerializer(SnowflakeModelSerializer):
    class Meta:
        model = MyModel
        fields = '__all__'

2. Write a Custom JsonRender and JsonParser

and do something like this with a lot of assumptions, spoiler bad programming

class SnowflakeJSONRenderer(JSONRenderer):

    def _convert_big_integers(self, obj):
        if isinstance(obj, dict):
            return {key: self._convert_big_integers(value) for key, value in obj.items()}
        elif isinstance(obj, list):
            return [self._convert_big_integers(item) for item in obj]
        elif isinstance(obj, int) and obj > 2**53:
            return str(obj)
        return obj

What im looking for

As far as i know there is no option to do this

REST_FRAMEWORK = {
'DEFAULT_MODEL_SERIALIZER_CLASS': 'path.to.utils.serializers.CustomModelSerializer',
}

Am i missing something? In your opinion what could be the best solution? I'm open to any suggestion

[ulgens]

I'd recommend going with the first option. It doesn't look pleasant at first, but it's easily maintainable if you add a Django system check to validate that the correct class is used.

Do you have a specific reason to use snowflake btw? If you are open to alternatives, I'd recommend UUID7 as a solid one.

[HoodyH]

Thx for the replay @ulgens

Actually we are also considering uuid, for sure it's a simpler implementation and probably a better compromise in our situation

I'm try to evaluate snowflake option for this reasons:

• url shorter so more easy to copy and read (and espose incremental ids is not the best option)
• smaller and faster in db than uuid
• distributed generation
• time based sorting, but in later releases
• i don't want to change again ids 😂

Probably it's overkill but i'm not committed to the switch yet

I've made some tests with this code and it's working, it's still a fast implementation.
I still need to consider:

• snowflake instance id sync, prob with redis that we alread have as cache
• time sync, in our case azure already do this

from django.db import models
from django.utils.encoding import smart_str
from rest_framework import serializers


class SnowflakeField(serializers.Field):
    """Convert snowflake data to string"""
    def to_representation(self, value):
        return smart_str(value)

    def to_internal_value(self, data):
        try:
            return int(data)
        except ValueError:
            raise serializers.ValidationError("Invalid integer value")


class SnowflakePrimaryKeyRelatedField(serializers.PrimaryKeyRelatedField, SnowflakeField):
    """Handle related primary keys as snowflake"""

    def to_representation(self, value):
        if self.pk_field is not None:
            return self.pk_field.to_representation(value.pk)
        return str(value.pk)

    def to_internal_value(self, data):
        data = super(SnowflakeField).to_representation(data)
        return super(serializers.PrimaryKeyRelatedField).to_internal_value(data)


class SnowflakeSlugRelatedField(serializers.SlugRelatedField):
    """Implement if used"""


class SnowflakeModelSerializer(serializers.ModelSerializer):
    """
    Snowflake is a bigint so convert to string to avoid frontend memory handling problems
    """

    serializer_field_mapping = {
        **serializers.ModelSerializer.serializer_field_mapping,
        models.BigAutoField: SnowflakeField,
        models.BigIntegerField: SnowflakeField
    }
    serializer_related_field = SnowflakePrimaryKeyRelatedField
    serializer_related_to_field = SnowflakeSlugRelatedField

My pk model

from django.db import models

class SnowflakeBigAutoField(models.BigAutoField):

    def pre_save(self, model_instance, add):
        if add and not getattr(model_instance, self.attname, None):
            value = self.get_next_value()
            setattr(model_instance, self.attname, value)

        return super().pre_save(model_instance, add)

    @staticmethod
    def get_next_value():
        return next(sf)

# settings.py
DEFAULT_AUTO_FIELD = 'core.snowflake.SnowflakeBigAutoField'

If we deoloy with snowfake i like to share the production code here, because i didn't find any useful resource about snowflake id with django

DEFAULT_MODEL_SERIALIZER_CLASS

In any case, is it possible in your opinion try to introduce this setting? I can submit a pull request.
I looked at the code and it looks possible, am I missing something?

[ulgens]

url shorter so more easy to copy and read (and espose incremental ids is not the best option)

You don't have to use or real primary keys for this. I could use UUID7 for primary keys, then some other unique identifier that's is shorter for URLs.

smaller and faster in db than uuid

I haven't yet come up with an application that makes a difference. It's emphasized in almost every benchmark, but personally, based on my real-world experience, there are tons of other small and easy-to-make mistakes that harm database performance much more than using UUIDs.

distributed generation
time based sorting

Both checked by UUID7: https://uuid7.com/

In any case, is it possible in your opinion try to introduce this setting?

It is, of course, possible, but I'm not sure that is the best way to approach the issue. My understanding is that you have a "Snowflake ID to JSON" conversion problem. Django model fields and their DRF equievalants are mapped by serializer_field_mapping: https://www.django-rest-framework.org/api-guide/serializers/#serializer_field_mapping Not having this dict configurable without customizing the class seems problematic to me, more problematic than not having the DEFAULT_MODEL_SERIALIZER_CLASS If serializer classes had a method to dynamically calculate that mapping, like

    def get_serializer_field_mapping(self):
        serializer_field_mapping = copy.deepcopy(self.serializer_field_mapping)

        extra_serializer_field_mapping = getattr(api_settings, 'EXTRA_SERIALIZER_FIELD_MAPPING', {})
        serializer_field_mapping.update(extra_serializer_field_mapping)

        return serializer_field_mapping

I'd support that PR.

[HoodyH]

then some other unique identifier that's is shorter for URLs

We are doing this right now almost everywhere except on some dashboard elements, but it's kinda a pain 🥲

At one point we also introduced the ability for the user to customize the slug to reach certain resources, it was more confusing than useful, so we switched back to autogeneration.

I like something like discord only one id in and out, no names no personalization no problems, it's ok even if uuid it's a bit longer. I don't see problems in this aproach, am i wrong?

Not having this dict configurable without customizing the class seems problematic to me

Next week i try to implemente this solution and let you know

My understanding is that you have a "Snowflake ID to JSON" conversion problem

There is actually a kind of bug: if you take a BigInt in DB, for example 2^64 and represent as int, JavaScript/TypeScript Number can only safely represent integers up to 2^53−1. JSON follows the same limitation, so although no error occurs, it is still technically incorrect.

It is possible to find a workaround on client side, but it is not guaranteed to work on all browsers.

[ulgens]

There is actually a kind of bug: if you take a BigInt in DB, for example 2^64 and represent as int, JavaScript/TypeScript Number can only safely represent integers up to 2^53−1. JSON follows the same limitation, so although no error occurs, it is still technically incorrect.

Sorry if I missed it but what's the result of this behaviour? Does it fail? Does it produce an incorrect response? If a Django native datatype can't be converted to a proper json by DRF, we may consider to add a warning and update the docs about it.