Field refactor (tortoise#206)

Split fields into many modules for categorisation
Remove unnecessary __slots__ from Fields, as this complicates work, and saves almost no memory as Fields only get instantiated as class variables (hence does not accumulate)
PyLint plugin now uses simplified Field-type inference and generates reverse FK's correctly.
Move DDL definition of fields from generate_schema into Fields
Make fields support overriding on a per-dialect level
Extend schema discover to know about DB types for ANY db type.
Make schema generator use types specified in Fields
Have fields define generated PK's
Have fields define custom emulation rules (e.g. DecimalField and SQLite)

Author: grigi

CHANGELOG.rst

@@ -3,8 +3,19 @@
 Changelog
 =========
 Unreleased
-------
-- Check whether charset name is available
+----------
+
+* Refactored Fields:
+
+  Fields have been refactored, for better maintenance. There should be no change for most users.
+
+  - More accurate auto-completion.
+  - Fields now contain their own SQL schema by dialect, which significantly simpliefies adding field types.
+  - ``describe_model()`` now returns the DB type, and dialect overrides.
+
+- ``JSONField`` will now automatically use ``python-rapidjson`` as an accelerator if it is available.
+- ``DecimalField`` and aggregations on it, now behaves much more like expected on SQLite (#256)
+- Check whether charset name is valid for the MySQL connection (#261)
 
 0.15.4
 ------

docs/examples/basic.rst

@@ -98,7 +98,7 @@ Recursive Relations
 
 .. _example_enum_fields:
 
-===============
+==================
 Enumeration Fields
-===============
+==================
 .. literalinclude::  ../../examples/enum_fields.py

docs/fields.rst

@@ -23,74 +23,29 @@ Fields are defined as properties of a ``Model`` class object:
 Reference
 =========
 
-.. autoclass:: tortoise.fields.Field
-    :members:
+Here is the list of fields available with custom options of these fields:
+
+Base Field
+----------
 
-Here is the list of fields available at the moment with custom options of these fields:
+.. automodule:: tortoise.fields.base
+    :members:
+    :undoc-members:
 
 Data Fields
 -----------
 
-.. autoclass:: tortoise.fields.IntField
-    :exclude-members: to_db_value, to_python_value
-
-.. autoclass:: tortoise.fields.BigIntField
-    :exclude-members: to_db_value, to_python_value
-
-.. autoclass:: tortoise.fields.SmallIntField
-    :exclude-members: to_db_value, to_python_value
-
-.. autoclass:: tortoise.fields.CharField
-    :exclude-members: to_db_value, to_python_value
-
-.. autoclass:: tortoise.fields.TextField
-    :exclude-members: to_db_value, to_python_value
-
-.. autoclass:: tortoise.fields.BooleanField
-    :exclude-members: to_db_value, to_python_value
-
-.. autoclass:: tortoise.fields.DecimalField
-    :exclude-members: to_db_value, to_python_value
-
-.. autoclass:: tortoise.fields.DatetimeField
-    :exclude-members: to_db_value, to_python_value
-
-.. autoclass:: tortoise.fields.DateField
-    :exclude-members: to_db_value, to_python_value
-
-.. autoclass:: tortoise.fields.TimeDeltaField
-    :exclude-members: to_db_value, to_python_value
-
-.. autoclass:: tortoise.fields.FloatField
-    :exclude-members: to_db_value, to_python_value
-
-.. autoclass:: tortoise.fields.JSONField
-    :exclude-members: to_db_value, to_python_value
-
-.. autoclass:: tortoise.fields.UUIDField
-    :exclude-members: to_db_value, to_python_value
-
-.. autoclass:: tortoise.fields.IntEnumField
-    :exclude-members: to_db_value, to_python_value
-
-.. autoclass:: tortoise.fields.CharEnumField
+.. automodule:: tortoise.fields.data
+    :members:
     :exclude-members: to_db_value, to_python_value
 
-
 Relational Fields
 -----------------
 
-.. autoclass:: tortoise.fields.ForeignKeyField
+.. automodule:: tortoise.fields.relational
+    :members: ForeignKeyField, OneToOneField, ManyToManyField
     :exclude-members: to_db_value, to_python_value
 
-.. autoclass:: tortoise.fields.OneToOneField
-
-.. autofunction:: tortoise.fields.ManyToManyField
-
-.. autodata:: tortoise.fields.ForeignKeyRelation
-
-.. autodata:: tortoise.fields.ForeignKeyNullableRelation
-
 Extending A Field
 =================
 

docs/getting_started.rst

@@ -26,6 +26,15 @@ Apart from ``asyncpg`` there is also support for ``sqlite`` through ``aiosqlite`
 ``mysql`` through ``aiomysql``.
 You can easily implement more backends if there is appropriate ``asyncio`` driver for this db.
 
+Optional Accelerators
+---------------------
+The following libraries can be used as accelerators:
+
+* `python-rapidjson <https://pypi.org/project/python-rapidjson/>`_: Automatically used if installed for JSON SerDes.
+* `uvloop <https://pypi.org/project/uvloop/>`_: Shown to improve performance, but needs to be set up.
+  Please look at ``uvloop`` docuemntation for more info.
+  If you use a framework, it may already use it.
+
 Tutorial
 ========
 

docs/query.rst

@@ -150,9 +150,21 @@ Foreign Key
 
 Tortoise ORM provides an API for working with FK relations
 
-.. autoclass:: tortoise.fields.ReverseRelation
+.. autoclass:: tortoise.fields.relational.ReverseRelation
     :members:
 
+.. autodata:: tortoise.fields.relational.ForeignKeyNullableRelation
+
+.. autodata:: tortoise.fields.relational.ForeignKeyRelation
+
+.. _one_to_one:
+
+One to One
+==========
+
+.. autodata:: tortoise.fields.relational.OneToOneNullableRelation
+
+.. autodata:: tortoise.fields.relational.OneToOneRelation
 
 .. _many_to_many:
 
@@ -161,7 +173,7 @@ Many to Many
 
 Tortoise ORM provides an API for working with M2M relations
 
-.. autoclass:: tortoise.fields.ManyToManyRelation
+.. autoclass:: tortoise.fields.relational.ManyToManyRelation
     :members:
     :inherited-members:
 

requirements.txt

@@ -1,6 +1,7 @@
 pypika>=0.35.8
 ciso8601>=2.1.1
 aiosqlite>=0.10.0
+typing-extensions>=3.7
+
+# For Python3.6 compatibility
 aiocontextvars>=0.2.2;python_version<"3.7"
-dataclasses>=0.6;python_version<"3.7"
-typing-extensions

setup.cfg

@@ -1,9 +1,9 @@
 [flake8]
 max-line-length = 100
 exclude =
-ignore = W503
 per-file-ignores =
     tortoise/__init__.py:F401
+    tortoise/fields/__init__.py:F401
     tests/schema/test_generate_schema.py:E501
 
 [isort]

tests/backends/__init__.py

@@ -0,0 +1,46 @@
+"""
+Test some mysql-specific features
+"""
+import ssl
+
+from tortoise import Tortoise
+from tortoise.contrib import test
+
+
+class TestMySQL(test.SimpleTestCase):
+    async def setUp(self):
+        if Tortoise._inited:
+            await self._tearDownDB()
+        self.db_config = test.getDBConfig(app_label="models", modules=["tests.testmodels"])
+        if self.db_config["connections"]["models"]["engine"] != "tortoise.backends.mysql":
+            raise test.SkipTest("MySQL only")
+
+    async def tearDown(self) -> None:
+        if Tortoise._inited:
+            await Tortoise._drop_databases()
+
+    async def test_bad_charset(self):
+        self.db_config["connections"]["models"]["credentials"]["charset"] = "terrible"
+        with self.assertRaisesRegex(ConnectionError, "Unknown charset"):
+            await Tortoise.init(self.db_config)
+
+    async def test_ssl_true(self):
+        self.db_config["connections"]["models"]["credentials"]["ssl"] = True
+        try:
+            await Tortoise.init(self.db_config)
+        except (ConnectionError, ssl.SSLError):
+            pass
+        else:
+            self.assertFalse(True, "Expected ConnectionError or SSLError")
+
+    async def test_ssl_custom(self):
+        # Expect connectionerror or pass
+        ctx = ssl.create_default_context()
+        ctx.check_hostname = False
+        ctx.verify_mode = ssl.CERT_NONE
+
+        self.db_config["connections"]["models"]["credentials"]["ssl"] = ctx
+        try:
+            await Tortoise.init(self.db_config, _create_db=True)
+        except ConnectionError:
+            pass

tests/test_capabilities.py tests/backends/test_capabilities.py

@@ -1,5 +1,5 @@
 """
-Test some postgres-specific features
+Test some PostgreSQL-specific features
 """
 import ssl
 
@@ -9,7 +9,7 @@
 from tortoise.exceptions import OperationalError
 
 
-class TestTwoDatabases(test.SimpleTestCase):
+class TestPostgreSQL(test.SimpleTestCase):
     async def setUp(self):
         if Tortoise._inited:
             await self._tearDownDB()

tests/test_connection_params.py tests/backends/test_connection_params.py

@@ -4,6 +4,7 @@
 from tortoise import fields
 from tortoise.contrib import test
 from tortoise.exceptions import ConfigurationError, IntegrityError
+from tortoise.functions import Avg, Max, Sum
 
 
 class TestDecimalFields(test.TestCase):
@@ -62,3 +63,53 @@ async def test_values_list(self):
             "decimal", "decimal_nodec"
         )
         self.assertEqual(list(values[0]), [Decimal("1.2346"), 19])
+
+    async def test_order_by(self):
+        await testmodels.DecimalFields.create(decimal=Decimal("0"), decimal_nodec=1)
+        await testmodels.DecimalFields.create(decimal=Decimal("9.99"), decimal_nodec=1)
+        await testmodels.DecimalFields.create(decimal=Decimal("27.27"), decimal_nodec=1)
+        values = (
+            await testmodels.DecimalFields.all()
+            .order_by("decimal")
+            .values_list("decimal", flat=True)
+        )
+        self.assertEqual(values, [Decimal("0"), Decimal("9.99"), Decimal("27.27")])
+
+    async def test_aggregate_sum(self):
+        await testmodels.DecimalFields.create(decimal=Decimal("0"), decimal_nodec=1)
+        await testmodels.DecimalFields.create(decimal=Decimal("9.99"), decimal_nodec=1)
+        await testmodels.DecimalFields.create(decimal=Decimal("27.27"), decimal_nodec=1)
+        values = (
+            await testmodels.DecimalFields.all()
+            .annotate(sum_decimal=Sum("decimal"))
+            .values("sum_decimal")
+        )
+        self.assertEqual(
+            values[0], {"sum_decimal": Decimal("37.26")},
+        )
+
+    async def test_aggregate_avg(self):
+        await testmodels.DecimalFields.create(decimal=Decimal("0"), decimal_nodec=1)
+        await testmodels.DecimalFields.create(decimal=Decimal("9.99"), decimal_nodec=1)
+        await testmodels.DecimalFields.create(decimal=Decimal("27.27"), decimal_nodec=1)
+        values = (
+            await testmodels.DecimalFields.all()
+            .annotate(avg_decimal=Avg("decimal"))
+            .values("avg_decimal")
+        )
+        self.assertEqual(
+            values[0], {"avg_decimal": Decimal("12.42")},
+        )
+
+    async def test_aggregate_max(self):
+        await testmodels.DecimalFields.create(decimal=Decimal("0"), decimal_nodec=1)
+        await testmodels.DecimalFields.create(decimal=Decimal("9.99"), decimal_nodec=1)
+        await testmodels.DecimalFields.create(decimal=Decimal("27.27"), decimal_nodec=1)
+        values = (
+            await testmodels.DecimalFields.all()
+            .annotate(max_decimal=Max("decimal"))
+            .values("max_decimal")
+        )
+        self.assertEqual(
+            values[0], {"max_decimal": Decimal("27.27")},
+        )

tests/test_db_url.py tests/backends/test_db_url.py

@@ -121,7 +121,7 @@ async def test_dup3_init(self):
 
     async def test_generated_nonint(self):
         with self.assertRaisesRegex(
-            ConfigurationError, "Generated primary key allowed only for IntField and BigIntField"
+            ConfigurationError, "Field 'val' \\(CharField\\) can't be DB-generated"
         ):
             await Tortoise.init(
                 {

tests/test_explain.py tests/backends/test_explain.py

@@ -14,7 +14,7 @@ cffi==1.13.2              # via cryptography
 chardet==3.0.4            # via requests
 ciso8601==2.1.2
 coverage==4.5.4           # via coveralls, pytest-cov
-coveralls==1.8.2
+coveralls==1.9.1
 cryptography==2.8         # via pymysql
 docopt==0.6.2             # via coveralls
 execnet==1.7.1            # via pytest-xdist
@@ -32,7 +32,7 @@ pytest-cov==2.8.1
 pytest-forked==1.1.3      # via pytest-xdist
 pytest-xdist==1.30.0
 pytest==5.3.1
-pyyaml==5.1.2
+pyyaml==5.2
 requests==2.22.0          # via coveralls
 six==1.13.0               # via cryptography, packaging, pytest-xdist
 typing-extensions==3.7.4.1

tests/backends/test_mysql.py

@@ -37,14 +37,17 @@ pytest-cov
 # Pypi
 twine<2.0
 
+# Optional accelerators
+uvloop>=0.12.0; sys_platform != "win32" and implementation_name == "cpython"
+python-rapidjson; implementation_name == "cpython"
+
 # Sample integration - Quart
 wsproto;python_version>="3.7"
 hypercorn;python_version>="3.7"
 quart;python_version>="3.7"
 
 # Sample integration - Sanic
 sanic
-uvloop>=0.12.0; sys_platform != "win32" and implementation_name == "cpython"
 
 # Sample integration - Starlette
 starlette

tests/test_postgres.py tests/backends/test_postgres.py

@@ -28,7 +28,7 @@ click==7.0                # via black, pip-tools, quart, uvicorn
 cloud-sptheme==1.10.0
 colorama==0.4.1           # via green
 coverage==4.5.4           # via coveralls, green, nose2, pytest-cov
-coveralls==1.8.2
+coveralls==1.9.1
 cryptography==2.8         # via pymysql
 docopt==0.6.2             # via coveralls
 docutils==0.15.2
@@ -49,7 +49,7 @@ hypercorn==0.9.0 ; python_version >= "3.7"
 hyperframe==5.2.0         # via h2
 idna==2.8                 # via httpcore, requests
 imagesize==1.1.0          # via sphinx
-importlib-metadata==0.23  # via pluggy, pytest, tox
+importlib-metadata==1.1.0  # via pluggy, pytest, tox
 isort==4.3.21             # via flake8-isort, pylint
 itsdangerous==1.1.0       # via quart
 jinja2==2.10.3            # via quart, sphinx
@@ -82,8 +82,9 @@ pytest-cov==2.8.1
 pytest-forked==1.1.3      # via pytest-xdist
 pytest-xdist==1.30.0
 pytest==5.3.1
+python-rapidjson==0.9.1 ; implementation_name == "cpython"
 pytz==2019.3              # via babel
-pyyaml==5.1.2
+pyyaml==5.2
 quart==0.10.0 ; python_version >= "3.7"
 readme-renderer==24.0     # via twine
 regex==2019.11.1          # via black
@@ -102,7 +103,7 @@ starlette==0.13.0
 stevedore==1.31.0         # via bandit
 testfixtures==6.10.3      # via flake8-isort
 toml==0.10.0              # via black, hypercorn, tox
-tox==3.14.1
+tox==3.14.2
 tqdm==4.40.0              # via twine
 twine==1.15.0
 typed-ast==1.4.0          # via astroid, black, mypy