Better schema generation, TruncationTestCase, values defaults (tortoise#162)

Features:
* .values() and .values_list() now default to all fields if none are specified.
* generate_schema() now generates well-formatted DDL SQL statements.
* Added TruncationTestCase testing class that truncates tables to allow faster testing of transactions. This allows tests to run 20% faster on SQLite, and about 7× faster for MySQL on my local system. (PG is 2×)
* Cleaned up the examples a bit
  * Re-did generate_schema example & turned into test case
  * A new recursive-lookup example & turned into test case

Bugfixes:
* Fixed state leak between database drivers which could cause incorrect DDL generation.
* Fixed missing table/column comment generation.
* Fixed comment generation to escape properly for SQLite
* Fixed comment generation for PostgreSQL to not duplicate comments

Chores:
* Minor tweaking to tests to run with one process to simplify stack traces
* Lots of small formatting/linting fixes
* Fixed requirements generation to work right on Py3.7
* Some documentation cleanup
* Sphinx 2.1 breaks badly with the documentation theme, and <2 currently gives the best result, so force it to an older version for the forseeable future.

Author: grigi

.gitignore

@@ -52,6 +52,7 @@ coverage.xml
 *,cover
 .hypothesis/
 .vscode/
+examples/*.sqlite3
 
 # Translations
 *.mo

.pylintrc

@@ -54,10 +54,9 @@ confidence=
 # --enable=similarities". If you want to run only the classes checker, but have
 # no Warning level messages displayed, use"--disable=all --enable=classes
 # --disable=W"
-disable=C0301,C0330,C0412,C0411,
-        W0613,W0212,W0611,
+disable=C0330,C0412,W0212,W0613,R0902,R0903,R0904,
         # tune down below here
-        R,C0111,C0103
+        C0103,C0111,R1702,R0912,R0913,R0914,R0915
 
 # Enable the message, report, category or checker with the given id(s). You can
 # either give multiple identifier separated by comma (,) or put this option

.travis.yml

@@ -12,11 +12,11 @@ env:
   - TORTOISE_TEST_DB="mysql://root:@127.0.0.1:3306/test_{}"
 matrix:
   include:
-  - python: pypy3.5
+  - python: pypy3.6-7.1.1
     env: TORTOISE_TEST_DB="sqlite:///tmp/test-{}.sqlite"
     install: pip install -r requirements-pypy.txt
     script: green
-  - python: pypy3.5
+  - python: pypy3.6-7.1.1
     env: TORTOISE_TEST_DB="mysql://root:@127.0.0.1:3306/test_{}"
     install: pip install -r requirements-pypy.txt
     script: green

CHANGELOG.rst

@@ -3,53 +3,83 @@
 Changelog
 =========
 
-0.12.9
-------
-* Example Sanic integration along with register_tortoise hook in contrib (#163)
-
-0.12.7
-------
-* Support connecting to PostgreSQL via Unix domain socket (simple case).
-* Self-referential Foreign and Many-to-Many keys are now allowed
-
-0.12.6
+0.13.0
 ------
+.. warning::
+   **This release brings with it, deprecation of Python 3.5:**
+
+   We will increase the minimum supported version of Python to 3.6,
+   as 3.5 is reaching end-of-life,
+   and is missing many useful features for async applications.
+
+   We will discontinue Python 3.5 support on the next major release (Likely 0.14.0)
+
+New Features:
+^^^^^^^^^^^^^
+- Example Sanic integration along with register_tortoise hook in contrib (#163)
+- ``.values()`` and ``.values_list()`` now default to all fields if none are specified.
+- ``generate_schema()`` now generates well-formatted DDL SQL statements.
+- Added ``TruncationTestCase`` testing class that truncates tables to allow faster testing of transactions.
+
+Bugfixes:
+^^^^^^^^^
+- Fixed state leak between database drivers which could cause incorrect DDL generation.
+- Fixed missing table/column comment generation for ``ForeignKeyField`` and ``ManyToManyField``
+- Fixed comment generation to escape properly for ``SQLite``
+- Fixed comment generation for ``PostgreSQL`` to not duplicate comments
+
+Docs/examples:
+^^^^^^^^^^^^^^
+- Examples have been reworked:
+
+  - Simplified init of many examples
+  - Re-did ``generate_schema.py`` example
+  - A new ``relations_recirsive.py`` example (turned into test case)
+
+- Lots of small documentation cleanups
+
+0.12.7 (retracted)
+------------------
+- Support connecting to PostgreSQL via Unix domain socket (simple case).
+- Self-referential Foreign and Many-to-Many keys are now allowed
+
+0.12.6 / 0.12.8
+---------------
 * Handle a ``__models__`` variable within modules to override the model discovery mechanism.
 
     If you define the ``__models__`` variable in ``yourapp.models`` (or wherever you specify to load your models from),
     ``generate_schema()`` will use that list, rather than automatically finding all models for you.
 
-* Split model consructor into from-Python and from-DB paths, leading to 15-25% speedup for large fetch operations.
-* More efficient queryset manipulation, 5-30% speedup for small fetches.
+- Split model consructor into from-Python and from-DB paths, leading to 15-25% speedup for large fetch operations.
+- More efficient queryset manipulation, 5-30% speedup for small fetches.
 
 0.12.5
 ------
-* Using non registered models or wrong references causes an ConfigurationError with a helpful message.
+- Using non registered models or wrong references causes an ConfigurationError with a helpful message.
 
 0.12.4
 ------
-* Inherit fields from Mixins, together with abstract model classes.
+- Inherit fields from Mixins, together with abstract model classes.
 
 0.12.3
 ------
-* Added description attribute to Field class. (#124)
-* Added the ability to leverage field description from (#124) to generate table column comments and ability to add table level comments
+- Added description attribute to Field class. (#124)
+- Added the ability to leverage field description from (#124) to generate table column comments and ability to add table level comments
 
 0.12.2
 ------
-* Fix accidental double order-by for ``.values()`` based queries. (#143)
+- Fix accidental double order-by for ``.values()`` based queries. (#143)
 
 0.12.1
 ------
-* Notable efficiency improvement for regular inserts
 * Bulk insert operation:
 
   .. note::
      The bulk insert operation will do the minimum to ensure that the object
      created in the DB has all the defaults and generated fields set,
-     but may be incomplete reference in Python.
+     this may result in incomplete references in Python.
 
-     e.g. ``IntField`` primary keys will not be poplulated.
+     e.g. ``IntField`` primary keys will not be populated.
 
   This is recommend only for throw away inserts where you want to ensure optimal
   insert performance.
@@ -61,6 +91,8 @@ Changelog
           User(name="...", email="...")
       ])
 
+- Notable efficiency improvement for regular inserts
+
 0.12.0
 ------
 * Tortoise ORM now supports non-autonumber primary keys.
@@ -69,7 +101,7 @@ Changelog
      This is a big feature change. It should not break any existing implementations.
 
   That primary key will be accesible through a reserved field ``pk`` which will be an alias of whichever field has been nominated as a primary key.
-  That alias field can be used as a field name when doing filtering e.g. ``.filter(pk=...)`` etc...
+  That alias field can be used as a field name when doing filtering e.g. ``.filter(pk=...)`` etc…
 
   We currently support single (non-composite) primary keys of any indexable field type, but only these field types are recommended:
 
@@ -138,13 +170,13 @@ Changelog
 
 0.11.4
 ------
-- Fixed several convenince issues with foreign relations:
+- Fixed several convenience issues with foreign relations:
 
   - FIXED: ``.all()`` actually returns the _query property as was documented.
   - New models with FK don't automatically fail to resolve any data. They can now be evaluated lazily.
 
 - Some DB's don't support OFFSET without Limit, added caps to signal workaround, which is to automatically add limit of 1000000
-- Pylint plugin to know about default `related_name` for ForeignKey fields.
+- Pylint plugin to know about default ``related_name`` for ForeignKey fields.
 - Simplified capabilities to be static, and defined at class level.
 
 0.11.3
@@ -161,19 +193,19 @@ Changelog
 
 * Added per-field indexes.
 
-  When setting `index=True` on a field, Tortoise will now generate an index for it.
+  When setting ``index=True`` on a field, Tortoise will now generate an index for it.
 
   .. note::
      Due to MySQL limitation of not supporting conditional index creation,
-     if `safe=True` (the default) is set, it won't create the index and emit a warning about it.
+     if ``safe=True`` (the default) is set, it won't create the index and emit a warning about it.
 
      We plan to work around this limitation in a future release.
 
 - Performance fix with PyPika for small fetch queries
 - Remove parameter hack now that PyPika support Parametrized queries
 - Fix typos in JSONField docstring
-- Added `.explain()` method on `QuerySet`.
-- Add `required` read-only property to fields
+- Added ``.explain()`` method on ``QuerySet``.
+- Add ``required`` read-only property to fields
 
 0.11.2
 ------
@@ -194,8 +226,8 @@ Changelog
 
 0.11.0
 ------
-- Added `.exclude()` method for QuerySet
-- Q objects can now be negated for `NOT` query (`~Q(...)`)
+- Added ``.exclude()`` method for QuerySet
+- Q objects can now be negated for ``NOT`` query (``~Q(...)``)
 - Support subclassing on existing fields
 - Numerous bug fixes
 - Removed known broken connection pooling

Makefile

@@ -18,6 +18,9 @@ help:
 up:
 	CUSTOM_COMPILE_COMMAND="make up" pip-compile -o requirements-pypy.txt requirements-pypy.in -U
 	CUSTOM_COMPILE_COMMAND="make up" pip-compile -o requirements-dev.txt requirements-dev.in -U
+	cat docs/extra_requirements.txt >> requirements-pypy.txt
+	cat docs/extra_requirements.txt >> requirements-dev.txt
+	sed -i "s/^-e .*/-e ./" requirements-dev.txt
 
 deps:
 	@pip install -q pip-tools

README.rst

@@ -30,7 +30,7 @@ You can find docs at `ReadTheDocs <http://tortoise-orm.readthedocs.io/en/latest/
    Tortoise ORM is young project and breaking changes are to be expected.
    We keep a `Changelog <http://tortoise-orm.readthedocs.io/en/latest/CHANGELOG.html>`_ and it will have possible breakage clearly documented.
 
-Tortoise ORM is supported on CPython >= 3.5.3 for SQLite, MySQL and PostgreSQL, and PyPy3.5 >= 5.10 for SQLite and MySQL only.
+Tortoise ORM is supported on CPython >= 3.5.3 for SQLite, MySQL and PostgreSQL, and PyPy3.6 >= 7.1 for SQLite and MySQL only.
 
 Why was Tortoise ORM built?
 ---------------------------

docs/CONTRIBUTING.rst

@@ -47,7 +47,7 @@ We have a ``Makefile`` that has the common operations listed, to get started jus
         docs        Builds the documentation
         style       Auto-formats the code
 
-So to run the tests you just need to run ``make test``, etc...
+So to run the tests you just need to run ``make test``, etc…
 
 The code is structured in the following directories:
 

docs/extra_requirements.txt

@@ -0,0 +1,3 @@
+aiocontextvars==0.2.2 ; python_version < "3.7"
+contextvars==2.4 ; python_version < "3.7"
+immutables==0.9 ; python_version < "3.7"

docs/index.rst

@@ -22,7 +22,7 @@ Why was Tortoise ORM built?
 ---------------------------
 
 Python has many existing and mature ORMs, unfortunately they are designed with an opposing paradigm of how I/O gets processed.
-``asyncio`` is relatively new technology that has a very different concurrency model, and the largest change is regarding how I/O is handled.
+``asyncio`` is relatively new technology that has a different concurrency model, and the largest change is regarding how I/O is handled.
 
 However, Tortoise ORM is not first attempt of building ``asyncio`` ORM, there are many cases of developers attempting to map synchronous python ORMs to the async world, initial attempts did not have a clean API.
 

docs/models.rst

@@ -82,7 +82,7 @@ Primary Keys
 In Tortoise ORM we require that a model has a primary key.
 
 That primary key will be accesible through a reserved field ``pk`` which will be an alias of whichever field has been nominated as a primary key.
-That alias field can be used as a field name when doing filtering e.g. ``.filter(pk=...)`` etc...
+That alias field can be used as a field name when doing filtering e.g. ``.filter(pk=...)`` etc…
 
 We currently support single (non-composite) primary keys of any indexable field type, but only these field types are recommended:
 
@@ -117,7 +117,7 @@ repetitive work by leveraging from inheritence.
 You can define fields in more generic classes and they are
 automatically available in derived classes. Base classes are
 not limited to Model classes. Any class will work. This way
-you are able to define your models in a very natural and easy
+you are able to define your models in a natural and easy
 to maintain way.
 
 Let's have a look at some examples.

docs/query.rst

@@ -116,7 +116,7 @@ produces 1 additional query, so ``.prefetch_related('events__participants')`` wi
 additional queries to fetch your data.
 
 Sometimes, when performance is crucial, you don't want to make additional queries like this.
-In cases like this you could use `values()` or `values_list()` to produce more efficient query
+In cases like this you could use ``values()`` or ``values_list()`` to produce more efficient query
 
 .. code-block:: python3
 

docs/requirements.txt

@@ -1,6 +1,6 @@
 -r ../requirements.txt
 
-sphinx
+sphinx < 2
 cloud_sptheme
-sphinx-autodoc-typehints
+sphinx-autodoc-typehints < 1.7
 asynctest

docs/roadmap.rst

@@ -11,13 +11,10 @@ For ``v1.0`` that involves:
 
 * Schema discovery
 * Timezone support
-* Clear and concise examples
-* Refactored Fields Schema generation
 * DB transaction isolation
 * Connection pooling
-* Robust connection handling
 * Change to all-parametrized queries for safety
-
+* Clear and concise examples
 
 Mid-term
 ========
@@ -26,7 +23,6 @@ Here we have all the features that is slightly further out, in no particular ord
 
 * Performance work:
     * Sub queries
-    * Bulk operations
     * Consider using Cython to accelerate critical loops
 
 * Convenience/Ease-Of-Use work:
@@ -43,15 +39,15 @@ Here we have all the features that is slightly further out, in no particular ord
     * Expand annotation framework to add statistical functions
 
 * Migrations
-    * Comprehensive schema in Migrations
+    * Comprehensive schema Migrations
     * Automatic forward Migration building
-    * Ability to easily run arb code in a migration
+    * Ability to easily run arbitrary code in a migration
     * Ability to get a the Models for that exact time of the migration, to ensure safe & consistent data migrations
     * Cross-DB support
     * Fixtures as a property of a migration
 
 * Serialization support
-    * Take inspiration from ``attrs`` and ``marshmallow``
+    * Take inspiration from ``attrs``, ``marshmallow`` and ``pydantic``
     * Provide sane default serializers that will work as-is for CRUD
     * Provide sane default schema generators
     * Make default serializers support some validation

examples/aggregation.py

@@ -1,6 +1,4 @@
-import asyncio
-
-from tortoise import Tortoise, fields
+from tortoise import Tortoise, fields, run_async
 from tortoise.aggregation import Count, Min, Sum
 from tortoise.models import Model
 
@@ -34,7 +32,7 @@ def __str__(self):
 
 
 async def run():
-    await Tortoise.init(config_file="config.json")
+    await Tortoise.init(db_url="sqlite://:memory:", modules={"models": ["__main__"]})
     await Tortoise.generate_schemas()
     tournament = await Tournament.create(name="New Tournament")
     await tournament.save()
@@ -60,5 +58,4 @@ async def run():
 
 
 if __name__ == "__main__":
-    loop = asyncio.get_event_loop()
-    loop.run_until_complete(run())
+    run_async(run())

examples/basic.py

@@ -1,9 +1,7 @@
 """
 This example demonstrates most basic operations with single model
 """
-import asyncio
-
-from tortoise import Tortoise, fields
+from tortoise import Tortoise, fields, run_async
 from tortoise.models import Model
 
 
@@ -20,7 +18,7 @@ def __str__(self):
 
 
 async def run():
-    await Tortoise.init(config_file="config.json")
+    await Tortoise.init(db_url="sqlite://:memory:", modules={"models": ["__main__"]})
     await Tortoise.generate_schemas()
 
     event = await Event.create(name="Test")
@@ -34,5 +32,4 @@ async def run():
 
 
 if __name__ == "__main__":
-    loop = asyncio.get_event_loop()
-    loop.run_until_complete(run())
+    run_async(run())