Skip to content

Commit 385b11b

Browse files
committed
docs(troubleshooting): copy in information that came up in the failure screnarios
https://docs.google.com/spreadsheets/d/1f1-fM7wJvxCUWvgzaq02Cyr_kPb5ubfCS64wnhZ5wWk/edit#gid=0
1 parent e1b3132 commit 385b11b

File tree

3 files changed

+27
-1
lines changed

3 files changed

+27
-1
lines changed

docs/configuration/environment-variables.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ A list of strings representing the host/domain names that this Django site can s
6161

6262
!!! warning "Deployment configuration"
6363

64-
You may change this setting when deploying the app to a non-localhost domain
64+
Do not enable this in production
6565

6666
!!! tldr "Django docs"
6767

docs/deployment/infrastructure.md

+2
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,8 @@ The following things in Azure are managed by the California Department of Techno
7878
- [Resource Groups](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/manage-resource-groups-portal)
7979
- Networking
8080
- Front Door
81+
- Web Application Firewall (WAF)
82+
- Distributed denial-of-service (DDoS) protection
8183
- IAM
8284
- Service connections
8385

docs/deployment/troubleshooting.md

+24
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,28 @@ If Terraform commands fail (locally or in the Pipeline) due to an `Error acquiri
6060
1. **Do any engineers have a Terrafrom command running locally?** You'll need to ask them. For example: They may have started an `apply` and it's sitting waiting for them to [approve](https://developer.hashicorp.com/terraform/cli/commands/apply#automatic-plan-mode) it. They will need to (gracefully) exit for the lock to be released.
6161
1. **If none of the steps above identified the source of the lock**, and especially if the `Created` time is more than ten minutes ago, that probably means the last Terraform command didn't release the lock. You'll need to grab the `ID` from the `Lock Info` output and [force unlock](https://developer.hashicorp.com/terraform/language/state/locking#force-unlock).
6262

63+
### App fails to start
64+
65+
If the container fails to start, you should see a [downtime alert](#monitoring). Assuming this app version was working in another [environment](../infrastructure/#environments), the issue is likely due to misconfiguration. Some things you can do:
66+
67+
- Check the [logs](#logs)
68+
- Ensure the [environment variables](../../configuration/environment-variables/) and [configuration data](../../configuration/data/) are set properly.
69+
- [Turn on debugging](../../configuration/environment-variables/#django_debug)
70+
- Force-push/revert the [environment](../infrastructure/#environments) branch back to the old version to roll back
71+
72+
### Littlepay API issue
73+
74+
Littlepay API issues may show up as:
75+
76+
- The [monitor](https://github.com/cal-itp/benefits/actions/workflows/check-api.yml) failing
77+
- The `Connect your card` button doesn't work
78+
79+
A common problem that causes Littlepay API failures is that the certificate expired. To resolve:
80+
81+
1. Reach out to <[email protected]>
82+
1. Receive a new certificate
83+
1. Put that certificate into the [configuration data](../../configuration/data/) and/or the [GitHub Actions secrets](https://github.com/cal-itp/benefits/settings/secrets/actions)
84+
6385
### Eligibility Server
6486

6587
If the Benefits application gets a 403 error when trying to make API calls to the [Eligibility Server](https://docs.calitp.org/eligibility-server/), it may be because the outbound IP addresses changed, and the Eligibility Server firewall is still restricting access to the old IP ranges.
@@ -70,3 +92,5 @@ If the Benefits application gets a 403 error when trying to make API calls to th
7092
1. Click `Edit`
7193
1. Click `Variables`
7294
1. Update the relevant variable with the new list of CIDRs
95+
96+
Note there is nightly downtime as the Eligibility Server restarts and loads new data.

0 commit comments

Comments
 (0)