docs(troulebshooting): separate the tools and specific issues sections

Author: afeld

docs/deployment/troubleshooting.md

@@ -1,14 +1,16 @@
 # Troubleshooting
 
-## Monitoring
+## Tools
+
+### Monitoring
 
 We have [ping tests](https://docs.microsoft.com/en-us/azure/azure-monitor/app/monitor-web-app-availability) set up to notify about availability of each [environment](../infrastructure/#environments). Alerts go to [#benefits-notify](https://cal-itp.slack.com/archives/C022HHSEE3F).
 
-## Logs
+### Logs
 
 Logs can be found a couple of places:
 
-### Azure App Service Logs
+#### Azure App Service Logs
 
 [Open the `Logs` for the environment you are interested in.](https://docs.google.com/document/d/11EPDIROBvg7cRtU2V42c6VBxcW_o8HhcyORALNtL_XY/edit#heading=h.6pxjhslhxwvj) The following tables are likely of interest:
 
@@ -18,7 +20,7 @@ Logs can be found a couple of places:
 
 For some pre-defined queries, click `Queries`, then `Group by: Query type`, and look under `Query pack queries`.
 
-### [Azure Monitor Logs](https://docs.microsoft.com/en-us/azure/azure-monitor/logs/data-platform-logs)
+#### [Azure Monitor Logs](https://docs.microsoft.com/en-us/azure/azure-monitor/logs/data-platform-logs)
 
 [Open the `Logs` for the environment you are interested in.](https://docs.google.com/document/d/11EPDIROBvg7cRtU2V42c6VBxcW_o8HhcyORALNtL_XY/edit#heading=h.n0oq4r1jo7zs)
 
@@ -31,19 +33,23 @@ In the latter two, you should see recent log output. Note [there is some latency
 
 See [`Failures`](https://docs.microsoft.com/en-us/azure/azure-monitor/app/asp-net-exceptions#diagnose-failures-using-the-azure-portal) in the sidebar (or `exceptions` under `Logs`) for application errors/exceptions.
 
-### Live tail
+#### Live tail
 
 After [setting up the Azure CLI](#making-changes), you can use the following command to [stream live logs](https://docs.microsoft.com/en-us/azure/app-service/troubleshoot-diagnostic-logs#in-local-terminal):
 
 ```sh
 az webapp log tail --resource-group RG-CDT-PUB-VIP-CALITP-P-001 --name AS-CDT-PUB-VIP-CALITP-P-001 2>&1 | grep -v /healthcheck
 ```
 
-### SCM
+#### SCM
 
 <https://as-cdt-pub-vip-calitp-p-001-dev.scm.azurewebsites.net/api/logs/docker>
 
-## Terraform lock
+## Specific issues
+
+This section serves the [runbook](https://www.pagerduty.com/resources/learn/what-is-a-runbook/) for Benefits.
+
+### Terraform lock
 
 [General info](https://developer.hashicorp.com/terraform/language/state/locking)
 
@@ -54,7 +60,7 @@ If Terraform commands fail (locally or in the Pipeline) due to an `Error acquiri
 1. **Do any engineers have a Terrafrom command running locally?** You'll need to ask them. For example: They may have started an `apply` and it's sitting waiting for them to [approve](https://developer.hashicorp.com/terraform/cli/commands/apply#automatic-plan-mode) it. They will need to (gracefully) exit for the lock to be released.
 1. **If none of the steps above identified the source of the lock**, and especially if the `Created` time is more than ten minutes ago, that probably means the last Terraform command didn't release the lock. You'll need to grab the `ID` from the `Lock Info` output and [force unlock](https://developer.hashicorp.com/terraform/language/state/locking#force-unlock).
 
-## Eligibility Server
+### Eligibility Server
 
 If the Benefits application gets a 403 error when trying to make API calls to the [Eligibility Server](https://docs.calitp.org/eligibility-server/), it may be because the outbound IP addresses changed, and the Eligibility Server firewall is still restricting access to the old IP ranges.