Document the new migration tool

Author: sandhose

docs/SUMMARY.md

@@ -33,6 +33,7 @@
     - [`database`](./reference/cli/database.md)
     - [`manage`](./reference/cli/manage.md)
     - [`server`](./reference/cli/server.md)
+    - [`syn2mas`](./reference/cli/syn2mas.md)
     - [`worker`](./reference/cli/worker.md)
     - [`templates`](./reference/cli/templates.md)
     - [`doctor`](./reference/cli/doctor.md)

docs/reference/cli/config.md

@@ -26,7 +26,7 @@ clients:
   # ...
 ```
 
-## `config generate`
+## `config generate [--synapse-config <synapse-config>] [--output <output>]`
 
 Generate a sample configuration file.
 It generates random signing keys (`.secrets.keys`) and the cookie encryption secret (`.secrets.encryption`).
@@ -38,6 +38,10 @@ INFO generate:rsa: mas_config::oauth2: Done generating RSA key
 INFO generate:ecdsa: mas_config::oauth2: Done generating ECDSA key
 ```
 
+The `--synapse-config` option can be used to migrate over configuration options from an existing Synapse configuration.
+
+The `--output` option can be used to specify the output file. If not specified, the output will be written to stdout.
+
 ## `config sync [--prune] [--dry-run]`
 
 Synchronize the configuration with the database.
@@ -52,4 +56,4 @@ INFO cli.config.sync: Updating provider provider.id=01H3FDH2XZJS8ADKRGWM84PZTY
 INFO cli.config.sync: Adding provider provider.id=01H3FDH2XZJS8ADKRGWM84PZTF
 INFO cli.config.sync: Deleting client client.id=01GFWRB9MYE0QYK60NZP2YF905
 INFO cli.config.sync: Updating client client.id=01GFWRB9MYE0QYK60NZP2YF904
-```
+```

docs/reference/cli/syn2mas.md

@@ -0,0 +1,29 @@
+# `syn2mas`
+
+Tool to import data from an existing Synapse homeserver into MAS.
+
+Global options:
+- `--config <config>`: Path to the MAS configuration file.
+- `--help`: Print help.
+- `--synapse-config <synapse-config>`: Path to the Synapse configuration file.
+- `--synapse-database-uri <synapse-database-uri>`: Override the Synapse database URI.
+
+## `syn2mas check`
+
+Check the setup for potential problems before running a migration
+
+```console
+$ mas-cli syn2mas check --config mas_config.yaml --synapse-config homeserver.yaml
+```
+
+## `syn2mas migrate [--dry-run]`
+
+Migrate data from the homeserver to MAS.
+
+The `--dry-run` option will perform a dry-run of the migration, which is safe to run without stopping Synapse.
+It will perform a full data migration, but then empty the MAS database at the end to roll back.
+
+
+```console
+$ mas-cli syn2mas migrate --config mas_config.yaml --synapse-config homeserver.yaml
+```

docs/setup/migration.md

@@ -1,29 +1,25 @@
 # Migrating an existing homeserver
 
-One of the design goals of MAS has been to allow it to be used to migrate an existing homeserver to an OIDC-based architecture.
+One of the design goals of MAS has been to allow it to be used to migrate an existing homeserver, specifically without requiring users to re-authenticate and ensuring that all existing clients continue to work.
 
-Specifically without requiring users to re-authenticate and that non-OIDC clients continue to work.
-
-Features that are provided to support this include:
+Features that support this include:
 
 - Ability to import existing password hashes from Synapse
 - Ability to import existing sessions and devices
-- Ability to import existing access tokens linked to devices (ie not including short-lived admin puppeted access tokens)
+- Ability to import existing access tokens
 - Ability to import existing upstream IdP subject ID mappings
 - Provides a compatibility layer for legacy Matrix authentication
 
-There will be tools to help with the migration process itself. But these aren't quite ready yet.
-
 ## Preparing for the migration
 
-The deployment is non-trivial so it is important to read through and understand the steps involved and make a plan before starting.
+The deployment is non-trivial, so it is important to read through and understand the steps involved and make a plan before starting.
 
 ### Is your setup ready to be migrated?
 
 #### SAML2 and LDAP Single Sign-On Providers are not supported
 
-A deployment which requires SAML or LDAP-based authentication should use a service like [Dex](https://github.com/dexidp/dex) to bridge between the SAML provider and the authentication service.
-MAS is different from Synapse in that it does **not** have built-in support for SAML or LDAP-based providers.
+A deployment that requires SAML or LDAP-based authentication should use a service like [Dex](https://github.com/dexidp/dex) to bridge between the SAML provider and the authentication service.
+MAS differs from Synapse in that it does **not** have built-in support for SAML or LDAP-based providers.
 
 #### Custom password providers are not supported
 
@@ -32,28 +28,34 @@ If your Synapse homeserver currently uses a custom password provider module, ple
 #### SQLite databases are not supported
 
 It is worth noting that MAS currently only supports PostgreSQL as a database backend.
+The migration tool only supports reading from PostgreSQL for the Synapse database as well.
 
 ### Install and configure MAS alongside your existing homeserver
 
 Follow the instructions in the [installation guide](installation.md) to install MAS alongside your existing homeserver.
 
 You'll need a blank PostgreSQL database for MAS to use; it does not share the database with the homeserver.
 
-Set up a configuration file but don't start MAS, or create any users, yet.
+MAS provides a tool to generate a configuration file based on your existing Synapse configuration. This is useful for kickstarting your new configuration.
+
+```sh
+mas-cli config generate --synapse-config homeserver.yaml --output mas_config.yaml
+```
+
+When using this tool, be careful to examine the log output for any warnings about unsupported configuration options.
 
 #### Local passwords
 
-Synapse uses bcrypt as its password hashing scheme while MAS defaults to using the newer argon2id.
+Synapse uses bcrypt as its password hashing scheme, while MAS defaults to using the newer argon2id.
 You will have to configure the version 1 scheme as bcrypt for migrated passwords to work.
-It is also recommended that you keep argon2id as version 2 so that once users log in, their hashes will be updated to the newer recommended scheme.
-If you have a `pepper` set in the `password_config` section of your Synapse config, then you need to specify this `pepper` as the `secret` field for your `bcrypt` scheme.
+It is also recommended that you keep argon2id as version 2 so that once users log in, their hashes will be updated to the newer, recommended scheme.
 
 Example passwords configuration:
 ```yml
 passwords:
   enabled: true
   schemes:
-  - version: 1 # TODO I think v:2 has to come first in this list
+  - version: 1
     algorithm: bcrypt
     # Optional, must match the `password_config.pepper` in the Synapse config
     #secret: secretPepperValue
@@ -70,22 +72,59 @@ The migration checker will inform you if this has not been configured properly.
 If you are using an upstream SSO provider, then you will need to configure the upstream provider in MAS manually.
 
 MAS does not support SAML or LDAP upstream providers.
-If you are using one of these, you will need to use an adapter such as Dex at this time,
-but we have not yet documented this procedure.
+If you are using one of these, you will need to use an adapter such as Dex at this time, but we have not yet documented this procedure.
 
 Each upstream provider that was used by at least one user in Synapse will need to be configured in MAS.
 
 Set the `synapse_idp_id` attribute on the provider to:
 
 - `"oidc"` if you used an OIDC provider in Synapse's legacy `oidc_config` configuration section.
-- `"oidc-myprovider"` if you used an OIDC provider in Synapse's `oidc_providers` configuration list,
-  with a `provider` of `"myprovider"`.
+- `"oidc-myprovider"` if you used an OIDC provider in Synapse's `oidc_providers` configuration list, with a `provider` of `"myprovider"`.
   (This is because Synapse prefixes the provider ID with `oidc-` internally.)
 
-Without the `synapse_idp_id`s being set, syn2mas does not understand which providers
-in Synapse correspond to which provider in MAS.
+Without the `synapse_idp_id`s being set, `mas-cli syn2mas` does not understand which providers in Synapse correspond to which provider in MAS.
+
+For example, if your Synapse configuration looked like this:
+
+```yaml
+oidc_providers:
+  - idp_id: dex
+    idp_name: "My Dex server"
+    issuer: "https://example.com/dex"
+    client_id: "synapse"
+    client_secret: "supersecret"
+    scopes: ["openid", "profile", "email"]
+    user_mapping_provider:
+      config:
+        localpart_template: "{{ user.email.split('@')[0].lower() }}"
+        email_template: "{{ user.email }}"
+        display_name_template: "{{ user.name|capitalize }}"
+```
 
-!!!!!!!!! TODO add an example here
+Then the equivalent configuration in MAS would look like this:
+
+```yaml
+upstream_oauth2:
+  providers:
+  - id: 01JSHPZHAXC50QBKH67MH33TNF
+    synapse_idp_id: oidc-dex
+    issuer: "https://example.com/dex"
+    human_name: "My Dex server"
+    client_id: "synapse"
+    client_secret: "supersecret"
+    token_endpoint_auth_method: client_secret_basic
+    scope: "email openid profile"
+    claims_imports:
+      localpart:
+        action: require
+        template: "{{ user.email.split('@')[0].lower() }}"
+      displayname:
+        action: force
+        template: "{{ user.name|capitalize }}"
+      email:
+        action: force
+        template: "{{ user.email }}"
+```
 
 The migration checker will inform you if a provider is missing from MAS' config.
 
@@ -95,58 +134,65 @@ You can use the `check` command of the `syn2mas` tool to identify configuration
 You do not need to stop Synapse to run this command.
 
 ```sh
-mas-cli --config mas_config.yaml syn2mas --synapse-config homeserver.yaml check
+mas-cli syn2mas check --config mas_config.yaml --synapse-config homeserver.yaml
 ```
 
-This will either output a list of errors and warnings, or tell you that the check completed with no errors or warnings.
+This will output a list of errors and warnings, or tell you that the check completed with no errors or warnings.
 
-If you have any errors, you must resolve these before starting the migration.
+If you have any errors, you must resolve them before starting the migration.
 
-If you have any warnings, please read, understand and possibly resolve them.
-With that said, resolving them is not strictly required before starting the migration.
+If you have any warnings, please read and understand them, and possibly resolve them.
+Resolving warnings is not strictly required before starting the migration.
 
 ### Do a dry-run of the import to test
 
-!!!!!!! TODO we don't have an exact dry-run mode exposed at the moment...
+MAS can perform a dry-run of the import, which is safe to run without stopping Synapse.
+It will perform a full data migration but then empty the MAS database at the end to roll back.
+
+This means it is safe to run multiple times without worrying about resetting the MAS database.
+It also means the time this dry-run takes is representative of the time it will take to perform the actual migration.
+
+```sh
+mas-cli syn2mas migrate --config mas_config.yaml --synapse-config homeserver.yaml --dry-run
+```
 
 ## Doing the migration
 
-Having done the preparation, you can now proceed with the actual migration. Note that this will require downtime for the homeserver and is not easily reversible.
+Having completed the preparation, you can now proceed with the actual migration. Note that this will require downtime for the homeserver and is not easily reversible.
 
 ### Backup your data and configuration
 
-As with any migration, it is important to backup your data before proceeding.
+As with any migration, it is important to back up your data before proceeding.
 
-We also suggest making a backup copy of your homeserver's known good configuration,
-before making any changes to enable MAS integration.
+We also suggest making a backup copy of your homeserver's known good configuration before making any changes to enable MAS integration.
 
-### Shutdown the homeserver
+### Shut down the homeserver
 
-This is to ensure that no new sessions are created whilst the migration is in progress.
+This ensures that no new sessions are created while the migration is in progress.
 
 ### Configure the homeserver to enable MAS integration
 
 Follow the instructions in the [homeserver configuration guide](homeserver.md) to configure the homeserver to use MAS.
 
 ### Do the import
 
-Once the homeserver has been stopped, MAS has been configured (but is not running!)
-and you have a successful migration check,
-run `syn2mas`'s `migrate` command.
-
-Other than the change of command word, the syntax is exactly the same as the `check` command.
+Once the homeserver has been stopped, MAS has been configured (but is not running!), and you have a successful migration check, run `syn2mas`'s `migrate` command.
 
 ```sh
-mas-cli --config mas_config.yaml syn2mas --synapse-config homeserver.yaml migrate
+mas-cli syn2mas migrate --config mas_config.yaml --synapse-config homeserver.yaml
 ```
 
 #### What to do if it goes wrong
 
 If the migration fails with an error:
 
-- You can either try to fix the error and make another attempt by re-running the command; or
-- you can revert your homeserver configuration (so MAS integration is disabled once more)
-  and abort the migration for now. In this case, you should not start MAS up.
+- You can try to fix the error and make another attempt by re-running the command; or
+- You can revert your homeserver configuration (so MAS integration is disabled once more) and abort the migration for now. In this case, you should not start MAS up.
+
+In *some cases*, MAS may have written to its own database during a failed migration, causing it to complain in subsequent runs.
+In this case, you can safely delete and recreate the MAS database, then start over.
+
+In *any case*, the migration tool itself **will not** write to the Synapse database, so as long as MAS hasn't been started, it is safe to roll back the migration without restoring the Synapse database.
 
 Please report migration failures to the developers.