Merge pull request #290 from conveyal/dev

Minor release

Author: evansiroky

README.md

@@ -155,6 +155,14 @@ SLACK_CHANNEL: '#devops'
 SLACK_WEBHOOK: https://hooks.slack.com/services/fake-code
 ```
 
+#### MS Teams Notifications
+
+To enable an MS Teams notification upon the completion (successful or not) of the deploy process, create a [MS Teams Webhook](https://docs.microsoft.com/en-us/microsoftteams/platform/concepts/connectors/connectors-using#setting-up-a-custom-incoming-webhook) and add the incoming webhook url as the `MS_TEAMS_WEBHOOK` key/value to your `env.yml`.
+
+```
+MS_TEAMS_WEBHOOK: https://outlook.office.com/webhook/123...abc
+```
+
 ### `flow`
 
 Run [Flow](https://flow.org/). Must have a `.flowconfig` in the current working directory and a `// @flow` annotation at the top of each file you want to check. See the Flow website for documentation.
@@ -199,7 +207,7 @@ $ mastarm prepublish lib:build
 
 ### `test`
 
-Run the [Jest](http://facebook.github.io/jest/) test runner on your project. By default, mastarm will run Jest and generate coverage reports on all .js files in the `lib` folder of your project. The `patterns` argument will make Jest run only tests whose filename match the provided pattern.
+Run the [Jest](http://facebook.github.io/jest/) test runner on your project.
 
 ```shell
 $ mastarm test
@@ -210,17 +218,51 @@ Run tests using Jest
 
 Options:
 
-  -h, --help                              output usage information
-  -u, --update-snapshots                  Force update of snapshots. USE WITH CAUTION.
-  --coverage                              Run Jest with coverage reporting
-  --coverage-paths <paths>                Extra paths to collect code coverage from
-  --jest-cli-args <args>                  Extra arguments to pass directly to the Jest Cli. Make sure to encapsulate all extra arguments in quote
-  --no-cache                              Run Jest without cache (defaults to using cache)
-  --run-in-band                           Run all tests serially in the current process
-  --setup-files <paths>                   Setup files to run before each test
-  --test-environment <env>                Jest test environment to use (Jest default is jsdom)
-  --test-path-ignore-patterns <patterns>  File patterns to ignore when scanning for test files
+-c, --config <path>                     Path to configuration files. (default: path.join(process.cwd() + '/configurations/default'))
+-e, --env <environment>                 Environment to use.
+-u, --update-snapshots                  Force update of snapshots. USE WITH CAUTION.
+--coverage                              Run Jest with coverage reporting
+--coverage-paths <paths>                Extra paths to collect code coverage from in addition to the mastarm default of `lib/**/*.js`
+--custom-config-file <path>             Override the Jest config with the values found in a file path relative to the current working directory
+--force-exit                            Force Jest to exit after all tests have completed running.
+--jest-cli-args <args>                  Extra arguments to pass directly to the Jest Cli. Make sure to encapsulate all extra arguments in quotes
+--no-cache                              Run Jest without cache (defaults to using cache)
+--run-in-band                           Run all tests serially in the current process. This is always set to true while running on in a continuous integration environment.
+--setup-files <paths>                   Setup files to run before each test
+--test-environment <env>                Jest test environment to use (Jest default is jsdom)
+--test-path-ignore-patterns <patterns>  File patterns to ignore when scanning for test files
+-h, --help                              output usage information
+
+```
+
+By default, mastarm will run Jest with most of the defaults in place. The defaults that mastarm adds include:
+
+- some transforms needed to read certain .js files and also YAML files.  
+- ignoring the test path directory `__tests__/test-utils`
+- setting the [testURL](https://jestjs.io/docs/en/configuration#testurl-string) to `http://localhost:9966`
+- turning on [notifications](https://jestjs.io/docs/en/configuration#notify-boolean) of test completion
+
+If the `coverage` flag is set to true, mastarm will automatically generate coverage reports of all .js files in the `lib` folder and will save the reports to the `coverage` folder.
 
+The `patterns` argument will make Jest run only tests whose filename match the provided pattern.
+
+There are a number of ways to set the [Jest config](https://jestjs.io/docs/en/configuration). The first is by adding a `jest` object to the package.json of the project. A number of other mastarm options will override the config. And finally, it is possible to use a custom config file (either .json or .js) via the `--custom-config-file` option. The config values are set and potentially overridden in the following order:
+
+1. mastarm defaults.
+2. Options in the `jest` object of the project's package.json file.
+3. The values specified in the mastarm arguments `--coverage-paths`, `--setup-files`, `--test-environment` and `--test-path-ignore-patterns`
+4. Options set in a custom config file specified in the mastarm argument `--custom-config-file`.
+
+Here is an example of how to set the config using a custom file:
+
+```shell
+mastarm test --custom-config-file __tests__/test-utils/mocks/mock-jest-config.json
+```
+
+It is also possible to override any [Jest CLI Options](https://jestjs.io/docs/en/cli) by setting the `--jest-cli-args` flag. Ex:
+
+```shell
+mastarm test --jest-cli-args "--json --outputFile e2e-test-results/results.json"
 ```
 
 ### `lint-messages`

__tests__/lib/__snapshots__/jest.js.snap

@@ -7,6 +7,7 @@ Array [
   "--runInBand",
   "--config",
   Object {
+    "collectCoverage": true,
     "collectCoverageFrom": Array [
       "lib/**/*.js",
       "bin",

__tests__/lib/jest.js

@@ -9,6 +9,7 @@ describe('jest.js', () => {
     const cfg = jestUtils.generateTestConfig(['these', 'files', 'only'], {
       cache: false,
       coveragePaths: 'bin src another-folder',
+      customConfigFile: '__tests__/test-utils/mocks/mock-jest-config.json',
       runInBand: true,
       setupFiles: 'beforeTestsSetup.js',
       testEnvironment: 'node',

__tests__/test-utils/mocks/mock-jest-config.json

@@ -0,0 +1 @@
+{ "collectCoverage": true }

bin/mastarm-deploy

@@ -3,11 +3,13 @@
 const path = require('path')
 
 const commander = require('commander')
+const execa = require('execa')
+const gitRepoIsUpToDate = require('git-repo-is-up-to-date')
 const commit = require('this-commit')()
 const username = require('username')
 
 const build = require('../lib/build')
-const {readFile} = require('../lib/fs-promise')
+const {readFile, writeFile} = require('../lib/fs-promise')
 const loadConfig = require('../lib/load-config')
 const logger = require('../lib/logger')
 const pkg = require('../lib/pkg')
@@ -28,74 +30,167 @@ commander
   .option('--s3bucket', 'S3 Bucket to push to.')
   .parse(process.argv)
 
-const url = pkg.repository.url.replace('.git', '')
-const tag = `<${url}/commit/${commit}|${pkg.name}@${commit.slice(0, 6)}>`
-const config = loadConfig(process.cwd(), commander.config, commander.env)
-const get = util.makeGetFn([commander, config.settings])
+// each of these variables are also used in the logToMsTeams function and
+// these need to be defined after potentially decoding a sops-encoded file
+let cloudfront, config, env, minify, s3bucket, tag, url
 
-if (config.env.SLACK_WEBHOOK && config.env.SLACK_WEBHOOK.length > 0) {
-  logger.logToSlack({
-    channel: config.env.SLACK_CHANNEL || '#devops',
-    webhook: config.env.SLACK_WEBHOOK
-  })
-}
+async function deploy () {
+  // get information about the directory that the config is in
+  const configRepoStatus = await gitRepoIsUpToDate(commander.config)
+  const { remoteUrl: configRemoteUrl, repoInfo } = configRepoStatus
+  let configCommit, configDir
+  if (repoInfo) {
+    configCommit = repoInfo.localCommit
+    configDir = repoInfo.root
+  }
 
-const files = util.parseEntries([...commander.args, ...(get('entries') || [])])
-util.assertEntriesExist(files)
-const sourceFiles = files.map(f => f[0])
-const outfiles = [...files.map(f => f[1]), ...files.map(f => `${f[1]}.map`)]
-
-const env = get('env') || 'development'
-const minify = get('minify')
-const buildOpts = {
-  config,
-  env,
-  files,
-  minify
-}
-const cloudfront = get('cloudfront')
-const s3bucket = get('s3bucket')
+  // do some extra analysis if it looks like a configurations repo is being used
+  if (configRemoteUrl && configRemoteUrl.endsWith('/configurations.git')) {
+    if (!configRepoStatus.isUpToDate) {
+      console.error('Configurations folder is not up-to-date! Errors:')
+      configRepoStatus.errors.forEach(err => console.error(err))
+      process.exit(1)
+    }
+
+    // decrypt env file using sops to make sure old file is overwritten with
+    // data from encoded sops file
+    const configPath = path.resolve(commander.config)
+    console.log('decrypting env file with sops')
+    const {stdout} = await execa(
+      'sops',
+      [
+        '-d',
+        path.join(configPath, 'env.enc.yml')
+      ]
+    )
+    await writeFile(path.join(configPath, 'env.yml'), stdout)
+    // at this point, we can be certain that the local configurations repo
+    // directory matches what has been committed and pushed to the remote repo
+  }
+
+  url = pkg.repository.url.replace('.git', '')
+  tag = `<${url}/commit/${commit}|${pkg.name}@${commit.slice(0, 6)}>`
+  config = loadConfig(process.cwd(), commander.config, commander.env)
+  const get = util.makeGetFn([commander, config.settings])
+
+  if (config.env.SLACK_WEBHOOK && config.env.SLACK_WEBHOOK.length > 0) {
+    logger.logToSlack({
+      channel: config.env.SLACK_CHANNEL || '#devops',
+      webhook: config.env.SLACK_WEBHOOK
+    })
+  }
 
-const pushToS3 = createPushToS3({
-  cloudfront,
-  s3bucket
-})
+  const files = util.parseEntries([...commander.args, ...(get('entries') || [])])
+  util.assertEntriesExist(files)
+  const sourceFiles = files.map(f => f[0])
+  const outfiles = [...files.map(f => f[1]), ...files.map(f => `${f[1]}.map`)]
 
-logger
-  .log(
+  env = get('env') || 'development'
+  minify = get('minify')
+  const buildOpts = {
+    config,
+    env,
+    files,
+    minify
+  }
+  cloudfront = get('cloudfront')
+  s3bucket = get('s3bucket')
+
+  const pushToS3 = createPushToS3({
+    cloudfront,
+    s3bucket
+  })
+
+  await logger.log(
     `:construction: *deploying: ${tag} by <@${username.sync()}>*
-:vertical_traffic_light: *mastarm:* v${mastarmVersion}
-:cloud: *cloudfront:* ${cloudfront}
-:hash: *commit:* ${commit}
-:seedling: *env:* ${env}
-:compression: *minify:* ${minify}
-:package: *s3bucket:* ${s3bucket}
-:hammer_and_wrench: *building:* ${sourceFiles.join(', ')}`
+    :vertical_traffic_light: *mastarm:* v${mastarmVersion}
+    :cloud: *cloudfront:* ${cloudfront}
+    :hash: *commit:* ${commit}
+    :seedling: *env:* ${env}
+    :compression: *minify:* ${minify}
+    :package: *s3bucket:* ${s3bucket}
+    :hammer_and_wrench: *building:* ${sourceFiles.join(', ')}`
   )
-  .then(() =>
-    build(buildOpts)
-      .then(() =>
-        logger.log(`:rocket: *uploading:* ${sourceFiles.length * 2} file(s)`)
-      )
-      .then(() =>
-        Promise.all(
-          outfiles.map(outfile =>
-            readFile(outfile).then(body => pushToS3({body, outfile}))
-          )
-        )
-      )
-      .then(() =>
-        logger
-          .log(
-            `:tada: :confetti_ball: :tada: *deploy ${tag} complete* :tada: :confetti_ball: :tada:`
-          )
-          .then(() => process.exit(0))
-      )
-      .catch(err =>
-        logger
-          .log(
-            `:rotating_light: *${tag} error deploying ${tag} ${err.message || err}*`
-          )
-          .then(() => process.exit(1))
+
+  try {
+    await build(buildOpts)
+    await logger.log(`:rocket: *uploading:* ${sourceFiles.length * 2} file(s)`)
+    await Promise.all(
+      outfiles.map(outfile =>
+        readFile(outfile).then(body => pushToS3({body, outfile}))
       )
-  )
+    )
+    await logger.log(
+      `:tada: :confetti_ball: :tada: *deploy ${tag} complete* :tada: :confetti_ball: :tada:`
+    )
+    await logToMsTeams({ configCommit, configDir, configRemoteUrl })
+    process.exit(0)
+  } catch (error) {
+    await logger.log(
+      `:rotating_light: *${tag} error deploying ${tag} ${error.message || error}*`
+    )
+    await logToMsTeams({ configCommit, configDir, configRemoteUrl, error })
+    process.exit(1)
+  }
+}
+
+deploy()
+
+/**
+ * Sends a card to MS Teams with information about the deployment
+ * @param  {[string]} configCommit   hash of the commit in the configurations
+ *                          repo (if it exists)
+ * @param  {[string]} configDir   partial path to specific config directory used
+ *                          to deploy
+ * @param  {[string]} configRemoteUrl base url for the configurations repo
+ *                          (if it exists)
+ * @param  {[Error]} error the error, if one occurred. A falsy value indicates
+ *                          success
+ */
+function logToMsTeams ({ configCommit, configDir, configRemoteUrl, error }) {
+  if (!config.env.MS_TEAMS_WEBHOOK) return Promise.resolve()
+
+  const potentialAction = [{
+    '@type': 'OpenUri',
+    name: `View Commit on Github`,
+    targets: [
+      {
+        os: 'default',
+        uri: `${url}/commit/${commit}`
+      }
+    ]
+  }]
+  if (configCommit && configRemoteUrl) {
+    potentialAction.push({
+      '@type': 'OpenUri',
+      name: `View Config Commit on Github`,
+      targets: [
+        {
+          os: 'default',
+          uri: `${configRemoteUrl}/tree/${configCommit}/${configDir}`
+        }
+      ]
+    })
+  }
+  const text = `📄 **commit:** ${pkg.name}@${commit.slice(0, 6)}\n
+  👤 **deployed by:** ${username.sync()}\n
+  ${configCommit
+    ? `🎛️ **config:** configurations@${configCommit.slice(0, 6)}\n
+  📂 **config folder:** ${configDir}\n` // improper indenting here needed to properly format on MS Teams
+    : '🎛️ **config:** unknown configuration data!\n'}
+  🚦 **mastarm:** v${mastarmVersion}\n
+  ☁️ **cloudfront:** ${cloudfront}\n
+  🌱 **env:** ${env}\n
+  🗜️ **minify:** ${minify}\n
+  📦 **s3bucket:** ${s3bucket}\n
+  ${error
+    ? `🚨 🚨 **error deploying ${error.message || error}**`
+    : `🎉 🎊 🎉 **deploy successful!** 🎉 🎊 🎉`}`
+
+  return logger.notifyMsTeams({
+    potentialAction,
+    text,
+    title: `${error ? 'Failed to deploy' : 'Successfully deployed'} ${pkg.name}`,
+    webhook: config.env.MS_TEAMS_WEBHOOK
+  })
+}

bin/mastarm-test

@@ -25,6 +25,10 @@ commander
     '--coverage-paths <paths>',
     'Extra paths to collect code coverage from'
   )
+  .option(
+    '--custom-config-file <path>',
+    'Override the Jest config with the values found in a file path relative to the current working directory'
+  )
   .option('--force-exit', 'Force Jest to exit after all tests have completed running.')
   .option(
     '--jest-cli-args <args>',

lib/eslintrc.json

@@ -8,13 +8,15 @@
     "standard",
     "standard-jsx",
     "plugin:flowtype/recommended",
-    "plugin:jest/recommended"
+    "plugin:jest/recommended",
+    "plugin:jsx-a11y/strict"
   ],
   "parser": "babel-eslint",
   "plugins": [
     "flowtype",
     "import",
-    "jest"
+    "jest",
+    "jsx-a11y"
   ],
   "rules": {
     "complexity": ["warn", 12],