fix #25 (#31)

* Change AutoFiled and SmallAutoField to clickhouse Int64, so that id worker can generate value for them.

* `DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'` is no longer a required configuration item.

Author: jayvynl

CHANGELOG.md

@@ -1,3 +1,8 @@
+### 1.1.0
+- Change `AutoFiled` and `SmallAutoField` to clickhouse `Int64`, so that id worker can generate value for them.
+This allows more compatibilities with existing apps such as `django.contrib.auth`.
+- `DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'` is no longer a required configuration item.
+
 ### 1.0.3
 - Fix reading settings in explain, pull request [#13](https://github.com/jayvynl/django-clickhouse-backend/pull/13) by [mahdi-jfri](https://github.com/mahdi-jfri).
 - Add toYYYYMM[DD[hhmmss]] functions.

README.md

@@ -28,6 +28,8 @@ Read [Documentation](https://github.com/jayvynl/django-clickhouse-backend/blob/m
 - In outer join, clickhouse will set missing columns to empty values (0 for number, empty string for text, unix epoch for date/datatime) instead of NULL. 
   So Count("book") resolve to 1 in a missing LEFT OUTER JOIN match, not 0.
   In aggregation expression Avg("book__rating", default=2.5), default=2.5 have no effect in a missing match.
+- Clickhouse does not support unique constraint and foreignkey constraint. `ForeignKey`, `ManyToManyField` and `OneToOneField` can be used with clickhouse backend, but no database level constraints will be added, so there could be some consistency problems.
+- Clickhouse does not support transaction. If any exception occurs during migrating, then your clickhouse database will be in an untracked state. Any migration should be full tested in test environment before deployed to production environment.
 
 **Requirements:**
 
@@ -85,13 +87,9 @@ DATABASES = {
         'HOST': 'localhost',
         'USER': 'DB_USER',
         'PASSWORD': 'DB_PASSWORD',
-        'TEST': {
-            'fake_transaction': True
-        }
     }
 }
 DATABASE_ROUTERS = ['dbrouters.ClickHouseRouter']
-DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
 ```
 
 ```python
@@ -143,8 +141,6 @@ automatically route your queries to the right database. In the preceding example
 queries from subclasses of `clickhouse_backend.models.ClickhouseModel` or custom migrations with a `clickhouse` hint key to clickhouse.
 All other queries are routed to the default database (postgresql).
 
-`DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'` is required to working with django migration.
-More details will be covered in [DEFAULT_AUTO_FIELD](https://github.com/jayvynl/django-clickhouse-backend/blob/main/docs/Configurations.md#default_auto_field).
 
 ### Model Definition
 
@@ -224,7 +220,7 @@ this operation will generate migration file under apps/migrations/
 then we mirgrate
 
 ```shell
-$ python manage.py migrate
+$ python manage.py migrate --database clickhouse
 ```
 
 for the first time run, this operation will generate django_migrations table with create table sql like this

clickhouse_backend/VERSION

@@ -1 +1 @@
-1.0.3
+1.1.0

clickhouse_backend/backend/base.py

@@ -23,8 +23,8 @@ class DatabaseWrapper(BaseDatabaseWrapper):
     # If a column type is set to None, it won't be included in the output.
     data_types = {
         # Django fields.
-        "SmallAutoField": "Int16",
-        "AutoField": "Int32",
+        "SmallAutoField": "Int64",
+        "AutoField": "Int64",
         "BigAutoField": "Int64",
         "IPAddressField": "IPv4",
         "GenericIPAddressField": "IPv6",

clickhouse_backend/backend/operations.py

@@ -23,8 +23,8 @@ class DatabaseOperations(BaseDatabaseOperations):
         "PositiveBigIntegerField": (0, 18446744073709551615),
         "PositiveSmallIntegerField": (0, 65535),
         "PositiveIntegerField": (0, 4294967295),
-        "SmallAutoField": (-32768, 32767),
-        "AutoField": (-2147483648, 2147483647),
+        "SmallAutoField": (-9223372036854775808, 9223372036854775807),
+        "AutoField": (-9223372036854775808, 9223372036854775807),
         "BigAutoField": (-9223372036854775808, 9223372036854775807),
         # Clickhouse fields.
         "Int8Field": (-1 << 7, -1 ^ (-1 << 7)),

clickhouse_backend/backend/schema.py

@@ -7,6 +7,7 @@
 from django.db.backends.ddl_references import (
     Expressions, IndexName, Statement, Table, Columns
 )
+from django.db.models.constraints import CheckConstraint, UniqueConstraint
 from django.db.models.expressions import ExpressionList
 from django.db.models.indexes import IndexExpression
 
@@ -82,9 +83,12 @@ def table_sql(self, model):
             ))
             constraints.append(self._column_check_sql(field))
         for constraint in model._meta.constraints:
-            constraints.append(
-                constraint.constraint_sql(model, self)
-            )
+            if isinstance(constraint, CheckConstraint):
+                constraints.append(
+                    constraint.constraint_sql(model, self)
+                )
+        if any(isinstance(c, UniqueConstraint) for c in model._meta.constraints):
+            warnings.warn("Clickhouse does not support unique constraint.")
 
         engine = self._get_engine(model)
         extra_parts = self._model_extra_sql(model, engine)

clickhouse_backend/compat.py

@@ -1,6 +1,6 @@
 import django
 
-dj32 = (3, 2) <= django.VERSION < (4, )
+dj3 = (3, ) <= django.VERSION < (4, )
 dj4 = (4, ) <= django.VERSION < (5, )
 dj_ge4 = django.VERSION >= (4, )
 dj_ge41 = django.VERSION >= (4, 1)

clickhouse_backend/models/sql/compiler.py

@@ -1,4 +1,4 @@
-from django.db.models.fields import BigAutoField
+from django.db.models.fields import AutoFieldMixin
 from django.db.models.sql import compiler
 
 from clickhouse_backend import compat
@@ -64,8 +64,8 @@ def as_sql(self):
         insert_statement = self.connection.ops.insert_statement()
 
         fields = self.query.fields
-        # For compatible with BigAutoField, add value generated by snowflake algorithm if needed.
-        absent_of_pk = isinstance(opts.pk, BigAutoField) and opts.pk not in fields
+        # Generate value for AutoField when needed.
+        absent_of_pk = isinstance(opts.pk, AutoFieldMixin) and opts.pk not in fields
         if absent_of_pk:
             fields = fields + [opts.pk]
             for obj in self.query.objs:

clickhouse_backend/patch/__init__.py

@@ -1,7 +1,15 @@
-from .fields import patch_jsonfield
-from .functions import patch_functions
+from .fields import *
+from .fields import __all__ as fields_all
+from .functions import *
+from .functions import __all__ as functions_all
+
+__all__ = [
+    "patch_all",
+    *fields_all,
+    *functions_all,
+]
 
 
 def patch_all():
-    patch_jsonfield()
-    patch_functions()
+    patch_all_functions()
+    patch_all_fields()

clickhouse_backend/patch/fields/__init__.py

@@ -0,0 +1,30 @@
+from django.db import models
+
+from .json import patch_jsonfield
+
+__all__ = [
+    "patch_all_fields",
+    "patch_auto_field",
+    "patch_jsonfield",
+]
+
+
+def patch_all_fields():
+    patch_auto_field()
+    patch_jsonfield()
+
+
+def patch_auto_field():
+    def rel_db_type_decorator(cls):
+        old_func = cls.rel_db_type
+
+        def rel_db_type(self, connection):
+            if connection.vendor == "clickhouse":
+                return self.db_type(connection)
+            return old_func(self, connection)
+
+        cls.rel_db_type = rel_db_type
+        return cls
+    rel_db_type_decorator(models.AutoField)
+    rel_db_type_decorator(models.SmallAutoField)
+    rel_db_type_decorator(models.BigAutoField)

clickhouse_backend/patch/fields.py renamed to clickhouse_backend/patch/fields/json.py

@@ -1,11 +1,18 @@
 from django.db.models import functions
 
+__all__ = [
+    "patch_all_functions",
+    "patch_random",
+]
 
-def random_as_clickhouse(self, compiler, connection, **extra_context):
-    return functions.Random.as_sql(
-        self, compiler, connection, function="rand64", **extra_context
-    )
 
+def patch_all_functions():
+    patch_random()
 
-def patch_functions():
+
+def patch_random():
+    def random_as_clickhouse(self, compiler, connection, **extra_context):
+        return functions.Random.as_sql(
+            self, compiler, connection, function="rand64", **extra_context
+        )
     functions.Random.as_clickhouse = random_as_clickhouse

clickhouse_backend/patch/functions.py

@@ -1,6 +1,6 @@
 services:
   clickhouse:
-    image: clickhouse/clickhouse-server:23.1.3.5
+    image: clickhouse/clickhouse-server:23.6.2.18
     container_name: test-clickhouse
     restart: always
     environment:

compose.yaml

@@ -51,9 +51,7 @@ Valid `TEST` keys:
   But this will have a side effect, that is, the clickhouse data of each test case will not be isolated. So in general it is not recommended to use this feature unless you know very well what is the impaction.
 
 
-### DEFAULT_AUTO_FIELD
-
-`DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'` IS REQUIRED TO WORKING WITH DJANGO MIGRATION.
+### Auto Field
 
 Django ORM depends heavily on single column primary key, this primary key is a unique identifier of an ORM object.
 All `get` `save` `delete` actions depend on primary key.
@@ -64,20 +62,10 @@ There is [no unique constraint](https://github.com/ClickHouse/ClickHouse/issues/
 
 By default, django will add a field named `id` as auto increasing primary key.
 
-- AutoField
-
-  Mapped to clickhouse Int32 data type. You should generate this unique id yourself.
-
-- BigAutoField
-
-  Mapped to clickhouse Int64 data type. If primary key is not specified when insert data, then `clickhouse_driver.idworker.id_worker` is used to generate this unique id.
-
-  Default id_worker is an instance of `clickhouse.idworker.snowflake.SnowflakeIDWorker` which implement [twitter snowflake id](https://en.wikipedia.org/wiki/Snowflake_ID).
-  If data insertions happen on multiple datacenter, server, process or thread, you should ensure uniqueness of (CLICKHOUSE_WORKER_ID, CLICKHOUSE_DATACENTER_ID) environment variable.
-  Because work_id and datacenter_id are 5 bits, they should be an integer between 0 and 31. CLICKHOUSE_WORKER_ID default to 0, CLICKHOUSE_DATACENTER_ID will be generated randomly if not provided.
+Django `AutoField`, `SmallAutoField` and `BigAutoField` ar mapped to clickhouse Int64 data type. If primary key is not specified when insert data, then `clickhouse_driver.idworker.id_worker` is used to generate this unique id.
 
-  `clickhouse.idworker.snowflake.SnowflakeIDWorker` is not thread safe. You could inherit `clickhouse.idworker.base.BaseIDWorker` and implement one, then set `CLICKHOUSE_ID_WORKER` in `settings.py` to doted import path of your IDWorker instance.
+Default `id_worker` is an instance of `clickhouse.idworker.snowflake.SnowflakeIDWorker` which implement [twitter snowflake id](https://en.wikipedia.org/wiki/Snowflake_ID).
+If data insertions happen on multiple datacenter, server, process or thread, you should ensure uniqueness of (CLICKHOUSE_WORKER_ID, CLICKHOUSE_DATACENTER_ID) environment variable.
+Because work_id and datacenter_id are 5 bits, they should be an integer between 0 and 31. CLICKHOUSE_WORKER_ID default to 0, CLICKHOUSE_DATACENTER_ID will be generated randomly if not provided.
 
-Django use a table named `django_migrations` to track migration files. ID field should be BigAutoField, so that IDWorker can generate unique id for you.
-After Django 3.2，a new [config `DEFAULT_AUTO_FIELD`](https://docs.djangoproject.com/en/4.1/releases/3.2/#customizing-type-of-auto-created-primary-keys) is introduced to control field type of default primary key.
-So `DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'` is required if you want to use migrations with django clickhouse backend.
+`clickhouse.idworker.snowflake.SnowflakeIDWorker` is not thread safe. You could inherit `clickhouse.idworker.base.BaseIDWorker` and implement one, then set `CLICKHOUSE_ID_WORKER` in `settings.py` to doted import path of your IDWorker instance.