Update our EDD process documentation (#166)

* Initial pass at updating our EDD database change processes

* Fix file name of new image

* Switch from 'rerunnable' to 'repeatable'

* Push the quick fixes from feedback

* Removed repeated use of Fowler's name as well as the repeated use of EDD

* Fix the image caption

* Removing all added personal pronouns

* Use markdown text styling

* Rename the application code version in the Phase definitions to be more clear

* Update terminology definitions

* Update language to be more focused

* Accepted introduction summary improvements

* Accepted suggested changes.

* Accept revised defenition and examples of non-destructive changes

* Update docs/contributing/database-migrations/edd.mdx

Co-authored-by: Thomas Avery <[email protected]>

* addtional updates

* revert to JSX to fix the broken image link

* Tweak the EDD doc (#184)

---------

Co-authored-by: Thomas Avery <[email protected]>
Co-authored-by: Oscar Hinton <[email protected]>

Author: 3 people

docs/contributing/database-migrations/edd.mdx

@@ -1,16 +1,21 @@
 import Tabs from "@theme/Tabs";
 import TabItem from "@theme/TabItem";
 
-# Evolutionary Database Design
+# Evolutionary database design
 
-At Bitwarden we follow
-[Evolutionary Database Design (EDD)](https://en.wikipedia.org/wiki/Evolutionary_database_design).
-EDD describes a process where the database schema is continuously updated while still ensuring
-compatibility with older releases by using database transition phases.
+At Bitwarden we follow [Evolutionary Database Design (EDD)][edd-wiki]. EDD describes a process where
+the database schema is continuously updated while still ensuring compatibility with older releases
+by defining a database transition phases.
 
-In short the Database Schema for the Bitwarden Server **must** support the previous release of the
-server. The database migrations will be performed before the code deployment, and in the event of a
-release rollback the database schema will **not** be updated.
+Bitwarden also needs to support:
+
+- **Zero-downtime deployments**: Which means that multiple versions of the application will be
+  running concurrently during the deployment window.
+- **Code rollback**: Critical defects in code should be able to be rolled back to the previous
+  version.
+
+To fulfill these additional requirements the database schema **must** support the previous release
+of the server.
 
 <bitwarden>
 
@@ -24,26 +29,76 @@ For background on this decision please see the [Evolutionary Database Design RFD
 
 ## Design
 
-### Nullable
+Database changes can be categorized into two categories: destructive and non-destructive changes
+\[[1](./edd#further-reading)\]. A destructive change prevents existing functionality from working as
+expected without an accompanying code change. A non-destructive change is the opposite: a database
+change that does not require a code change to allow the non-application to continue working as
+expected.
+
+### Non-destructive changes
+
+Many database changes can be designed in a backwards compatible manner by using a mix of nullable
+fields and default values in the database tables, views, and stored procedures. This ensures that
+the stored procedures can be called without the new columns and allow them to run with both the old
+and new code.
+
+### Destructive changes
+
+Any change that cannot be done in a non-destructive manner is a destructive change. This can be as
+simple as adding a non nullable column where the value needs to be computed from existing fields, or
+renaming an existing column. To handle destructive changes it's necessary to break them up into
+three phases: _Start_, _Transition_, and _End_ as shown in the diagram below.
+
+<figure>
+
+![Refactoring Stages](./transitions.png)
+
+<figcaption>Refactoring Phases</figcaption>
+
+</figure>
+
+It's worth noting that the _Refactoring Phases_ are usually rolling, and the _End phase_ of one
+refactor is the _Transition phase_ of another. The table below details which application releases
+needs to be supported during which database phase.
 
-Database tables, views and stored procedures should almost always use either nullable fields or have
-a default value. Since this will allow stored procedures to omit columns, which is a requirement
-when running both old and new code.
+| Database Phase | Release X | Release X+1 | Release X+2 |
+| -------------- | --------- | ----------- | ----------- |
+| Start          | ✅        | ❌          | ❌          |
+| Transition     | ✅        | ✅          | ❌          |
+| End            | ❌        | ✅          | ✅          |
 
-### EDD Process
+### Migrations
 
-The EDD breaks up each database migration into three phases. _Start_, _Transition_ and _End_.
+The three different migrations described in the diagram above are, _Initial migration_, _Transition
+migration_ and _ Finalization migration_.
 
-![Refactoring Stages](./stages_refactoring.jpg)
-[https://www.martinfowler.com/articles/evodb.html#TransitionPhase](https://www.martinfowler.com/articles/evodb.html#TransitionPhase)
+#### Initial migration
 
-This necessitates two different database migrations. The first migration adds new content and is
-backwards compatible with the existing code. The second migration removes content and is not
-backwards compatible with that same code prior to the first migration.
+The initial migration runs before the code deployment, and its purpose is to add support for
+_Release X+1_ without breaking support of _Release X_. The migration should execute quickly and not
+contain any costly operations to ensure zero downtime.
+
+#### Transition migration
+
+The transition migration are run sometime during the transition phase, and provides an optional data
+migration should it be too slow or put too much load on the database, or otherwise make it
+unsuitable for the _Initial migration_.
+
+- Compatible with _Release X_ **and** _Release X+1_ application.
+- Only data population migrations may be run at this time, if they are needed
+  - Must be run as a background task during the Transition phase.
+  - Operation is batched or otherwise optimized to ensure the database stays responsive.
+- Schema changes are NOT to be run during this phase.
+
+#### Finalization migration
+
+The finalization migration removes the temporary measurements that were needed to retain backwards
+compatibility with _Release X_, and the database schema henceforth only supports _Release X+1_.
+These migrations are run as part of the deployment of _Release X+2_.
 
 ### Example
 
-Let’s look at an example, the rename column refactor is shown in the image below.
+Let's look at an example, the rename column refactor is shown in the image below.
 
 ![Rename Column Refactor](./rename-column.gif)
 
@@ -73,7 +128,7 @@ actions.
 :::
 
 <Tabs>
-<TabItem value="first" label="First Migration" default>
+<TabItem value="first" label="Initial Migration" default>
 
 ```sql
 -- Add Column
@@ -120,7 +175,7 @@ END
 ```
 
 </TabItem>
-<TabItem value="data" label="Data Migration">
+<TabItem value="data" label="Transition Migration">
 
 ```sql
 UPDATE [dbo].Customer SET
@@ -129,7 +184,7 @@ WHERE FirstName IS NULL
 ```
 
 </TabItem>
-<TabItem value="second" label="Second Migration">
+<TabItem value="second" label="Finalization Migration">
 
 ```sql
 -- Remove Column
@@ -173,65 +228,96 @@ END
 </TabItem>
 </Tabs>
 
-## Workflow
+## Deployment orchestration
+
+There are some important constraints to the implementation of the process:
+
+- Bitwarden Production environments are required to be on at all times
+- Self-host instances must support the same database change process; however, they do not have the
+  same always-on application constraint
+- Minimization of manual steps in the process
+
+The process to support all of these constraints is a complex one. Below is an image of a state
+machine that will hopefully help visualize the process and what it supports. It assumes that all
+database changes follow the standards that are laid out in [Migrations](./).
+
+---
+
+![Bitwarden EDD State Machine](./edd_state_machine.jpg) \[Open Image in a new tab for better
+viewing\]
+
+---
 
-The Bitwarden specific workflow for writing migrations are described below.
+### Online environments
 
-### Developer
+Schema migrations and data migrations as just migrations. The underlying implementation issue is
+orchestrating the runtime constraints on the migration. Eventually, all migrations will end up in
+`DbScripts`. However, to orchestrate the running of _Transition_ and associated _Finalization_
+migrations, they are kept outside of `DbScripts` until the correct timing.
 
-The development flow is described in [Migrations](./).
+In environments with always-on applications, _Transition_ scripts must be run after the new code has
+been rolled out. To execute a full deploy, all new migrations in `DbScripts` are run, the new code
+is rolled out, and then all _Transition_ migrations in the `DbScripts_transition` directory are run
+as soon as all of the new code services are online. In the case of a critical failure after the new
+code is rolled out, a Rollback would be conducted (see Rollbacks below). _Finalization_ migrations
+will not be run until the start of the next deploy when they are moved into `DbScripts`.
 
-### Devops
+After this deploy, to prep for the next release, all migrations in `DbScripts_transition` are moved
+to `DbScripts` and then all migrations in `DbScripts_finalization` are moved to `DbScripts`,
+conserving their execution order for a clean install. For the current branching strategy, PRs will
+be open against `master` when `rc` is cut to prep for this release. This PR automation will also
+handle renaming the migration file and updating any reference of `[dbo_finalization]` to `[dbo]`.
 
-#### On `rc` cut
+The next deploy will pick up the newly added migrations in `DbScripts` and set the previously
+repeatable _Transition_ migrations to no longer be repeatable, execute the _Finalization_
+migrations, and then execute any new migrations associated with the code changes that are about to
+go out.
 
-Create a PR moving the future scripts.
+The state of migrations in the different directories at any one time is is saved and versioned in
+the Migrator Utility which supports the phased migration process in both types of environments.
 
-- `DbScripts_future` to `DbScripts`, prefix the script with the current date, but retain the
-  existing date.
-- `dbo_future` to `dbo`.
-  <bitwarden>
-    <li>
-      Create a ticket in Jira with a `Due Date` of the release date to ensure future migrations are
-      merged in and ready to be executed. Set the ticket that created the future migration as a
-      blocker.
-    </li>
-  </bitwarden>
+### Offline environments
 
-#### After server release
+The process for offline environments is similar to the always-on ones. However, since they do not
+have the constraint of always being on, the _Initial_ and _Transition_ migrations will be run one
+after the other:
 
-1. Run whatever data migration scripts might be needed. (This might need to be batched and executed
-   until all the data has been migrated)
-2. After having the server run for a while execute the future migration script to clean up the
-   database.
+- Stop the Bitwarden stack as done today
+- Start the database
+- Run all new migrations in `DbScripts` (both _Finalization_ migrations from the last deploy and any
+  _Initial_ migrations from the deploy currently going out)
+- Run all _Transition_ migrations
+- Restart the Bitwarden stack.
 
 ## Rollbacks
 
 In the event the server release failed and needs to be rolled back, it should be as simple as just
 re-deploying the previous version again. The database will **stay** in the transition phase until a
-hotfix can be released, and the server can be updated.
+patch can be released, and the server can be updated. Once a patch is ready to go out, it is
+deployed the _Transition_ migrations are rerun to verify that the DB is in the state that it is
+required to be in.
 
-The goal is to resolve the issue quickly and re-deploy the fixed code to minimize the time the
-database stays in the transition phase. Should a feature need to be completely pulled, a new
-migration needs to be written to undo the database changes and the future migration will also need
-to be updated to work with the database changes. This is generally not recommended since pending
-migrations (for other releases) will need to be revisited.
+Should a feature need to be completely pulled, a new migration needs to be written to undo the
+database changes and the future migration will also need to be updated to work with the database
+changes. This is generally not recommended since pending migrations (for other releases) will need
+to be revisited.
 
 ## Testing
 
 Prior to merging a PR please ensure that the database changes run well on the currently released
 version. We currently do not have an automated test suite for this and it’s up to the developers to
 ensure their database changes run correctly against the currently released version.
 
-## Further Reading
+## Further reading
 
-- [Evolutionary Database Design](https://martinfowler.com/articles/evodb.html) (Particularly
-  [All database changes are database refactorings](https://martinfowler.com/articles/evodb.html#AllDatabaseChangesAreMigrations))
-- [The Agile Data (AD) Method](http://agiledata.org/) (Particularly
-  [Catalog of Database Refactorings](http://agiledata.org/essays/databaseRefactoringCatalog.html))
-- [Refactoring Databases: Evolutionary Database](https://databaserefactoring.com/)
-- Refactoring Databases: Evolutionary Database Design (Addison-Wesley Signature Series (Fowler))
-  ISBN-10: 0321774515
+1. [Evolutionary Database Design](https://martinfowler.com/articles/evodb.html) (Particularly
+   [All database changes are database refactorings](https://martinfowler.com/articles/evodb.html#AllDatabaseChangesAreMigrations))
+2. [The Agile Data (AD) Method](http://agiledata.org/) (Particularly
+   [Catalog of Database Refactorings](http://agiledata.org/essays/databaseRefactoringCatalog.html))
+3. [Refactoring Databases: Evolutionary Database](https://databaserefactoring.com/)
+4. Refactoring Databases: Evolutionary Database Design (Addison-Wesley Signature Series (Fowler))
+   ISBN-10: 0321774515
 
+[edd-wiki]: https://en.wikipedia.org/wiki/Evolutionary_database_design
 [edd-rfd]:
   https://bitwarden.atlassian.net/wiki/spaces/PIQ/pages/177701412/Adopt+Evolutionary+database+design

docs/contributing/database-migrations/edd_state_machine.jpg

@@ -2,9 +2,9 @@
 sidebar_position: 2
 ---
 
-# Database Migrations
+# Database migrations
 
-## Applying Migrations
+## Applying migrations
 
 We use a `migrate.ps1` PowerShell script to apply migrations to the local development database. This
 script handles the different database providers that we support.
@@ -13,12 +13,12 @@ For instructions on how to use `migrate.ps1`, see the Getting Started section fo
 [MSSQL](../../getting-started/server/database/mssql/index.md#updating-the-database) and
 [Entity Framework](../../getting-started/server/database/ef/index.mdx#migrations)
 
-## Creating Migrations for New Changes
+## Creating migrations for new changes
 
 Any database change must be scripted as a migration for both our primary DBMS - MSSQL - as well as
 for Entity Framework. Follow the instructions below for each provider.
 
-### MSSQL Migrations
+### MSSQL migrations
 
 :::tip
 
@@ -37,24 +37,24 @@ It is possible that a change may not require a non-backwards-compatible end phas
 may be backwards-compatible in their final form). In that case, only one phase of changes is
 required.
 
-#### Backwards Compatible Migration
+#### Backwards compatible migration
 
 1. Modify the source `.sql` files in `src/Sql/dbo`.
 2. Write a migration script, and place it in `util/Migrator/DbScripts`. Each script must be prefixed
    with the current date.
 
-#### Non-Backwards Compatible Migration
+#### Non-backwards compatible migration
 
-1. Copy the relevant `.sql` files from `src/Sql/dbo` to `src/Sql/dbo_future`.
+1. Copy the relevant `.sql` files from `src/Sql/dbo` to `src/Sql/dbo_finalization`.
 2. Remove the backwards compatibility that is no longer needed.
-3. Write a new Migration and place it in `src/Migrator/DbScripts_future`. Name it
-   `YYYY-0M-FutureMigration.sql`.
+3. Write a new Migration and place it in `src/Migrator/DbScripts_finalization`. Name it
+   `YYYY-0M-FinalizationMigration.sql`.
    - Typically migrations are designed to be run in sequence. However since the migrations in
      DbScripts_future can be run out of order, care must be taken to ensure they remain compatible
      with the changes to DbScripts. In order to achieve this we only keep a single migration, which
      executes all backwards incompatible schema changes.
 
-### EF Migrations
+### EF migrations
 
 If you alter the database schema, you must create an EF migration script to ensure that EF databases
 keep pace with these changes. Developers must do this and include the migrations with their PR.
@@ -72,4 +72,15 @@ pwsh ef_migrate.ps1 [NAME_OF_MIGRATION]
 
 This will generate the migrations, which should then be included in your PR.
 
+### [Not Yet Implemented] Manual MSSQL migrations
+
+There may be a need for a migration to be run outside of our normal update process. These types of
+migrations should be saved for very exceptional purposes. One such reason could be an Index rebuild.
+
+1. Write a new Migration with a prefixed current date and place it in
+   `src/Migrator/DbScripts_manual`
+2. After it has been run against our Cloud environments and we are satisfied with the outcome,
+   create a PR to move it to `DbScripts`. This will enable it to be run by our Migrator processes in
+   self-host and clean installs of both cloud and self-host environments
+
 [code-style-sql]: ../code-style/sql.md