OSPO Infrastructure Management

This repository contains Ansible playbooks and roles for deploying and managing infrastructure for Augur and 8Knot services in production (EC2 instances).

Overview

Ansible is a powerful automation tool for managing and configuring server infrastructure, eliminating the need for manual setup and configuration of potentially many servers. In Ansible you define changes to the system by asserting what you want the result to be. This allows ansible to only make changes if they are needed. These assertions are stored in simple YAML files and can be organized into "playbooks", "roles", and "tasks" to help automate anything from large processes, small categories and individual services, or specific actions.

Configuration Structure

Playbooks

This repository is organized with all the high-level processes defined in playbooks within the playbooks folder. This includes things like installing 8Knot, installing augur, upgrading a system's apt packages, performing backups, etc.

See each individual .yml file in this folder to read more about what it is for.

Roles and Tasks In order to improve maintainability and reuse of common processes, these playbooks refer to "roles", which are a slightly more granular way to group things in ansible. These are located in the roles folder and contain various configuration items for the different services (augur, 8knot, etc).

Within these roles there are files that define specific tasks. These files are where the actual definitions for the actions being performed usually live, broken up granularly enough to allow them to be reused by multiple playbooks or even other tasks as needed.

Inventory The inventory/inventory.ini file defines which machines are available to run ansible playbooks against. These can be grouped for more easily identifying multiple machines to apply these same processes to (if we wanted to host more than one of each service).

Configuration the ansible.cfg file defines configuration items that help ansible know where to find things, such as the roles and inventory files, so that using ansible can be easier and more automatic. While you can put configuration specific to your machine in there, it would be preferred to put these items in ~/.ansible.cfg or some other config location since this config file is shared by everyone who uses these ansible scripts.

Quick Start

Prerequisites

1. Ansible installed on your control machine (i.e. the one you are probably reading this from)
2. AWS credentials configured (for EBS operations)
3. Vault password for encrypted secrets
4. SSH access to target servers

Setup

Many parts of this script require access to secrets. Some of these (like your SSH key) may already be set up for allowing you to access the services, others, like AWS access and the ansible vault password, are things you may need additional approval to access.

The main way secrets are stored in this repository is using ansible vault - this encrypts any secret values such that they can only be read (and thus used in playbooks) when you have the password. To learn more about using ansible vault, check out this digitalocean primer.

Usage

provided all the prerequisites are present, usage should be a relatively simple matter of running ansible-playbook followed by the path to the playbook you want to run (from the playbooks folder).

However, depending on the playbook you may need extra parameters:

• --ask-vault-pass - If the playbook you are running requires accessing secret encrypted variables, this will prompt you for the password to decrypt those variables.
• --ask-become-pass - If the server you are connecting to asks for a password the first time you use sudo, you need to provide that password so ansible can elevate its permissions when needed.
• -e "ansible_user=username" - If your username on the relevant system is different from the one on your local machine, ansible needs to know what it is so it can log in.

Additional, less recommended options

• --vault-id - this tells ansible which credentials to use for ansible vault. The value prod is most likely what you want. Note that depending on how you store your ansible vault credentials, you may need to specify something else, such as:
  • prod@<path to script> if you use a script to fetch your ansible vault password

Example

for example, following all the above guidelines, kicking off a snapshot of the Augur DB would look like this ansible-playbook playbooks/augur-clonedb.yml --ask-vault-pass --ask-become-pass -e "ansible_user=username"

Included Module Testing

This ansible directory comes with a custom module for fetching the AWS volume ID from a provided device file.

To test this module in a one-off setting, run:

ansible -m identify_volume -a 'partition_device=/dev/whatever' [somehost]

This relies on ansible being set to use the library directory for modules. This is present in ansible.cfg already, but if not you can add ANSIBLE_LIBRARY=./library to the start of the above command.

This module is designed to simply gather information using the fact that AWS sets disk serial numbers such that they align with the AWS volume identifiers. It is not destructive.