*** PROBLEM *** aap.coe.muc.redhat.com / Check expired date of certificate is WARNING

[rbo]

--SERVICE-ALERT-------------------
-
- Hostaddress: 10.32.98.2
- Hostname:    [aap.coe.muc.redhat.com](http://aap.coe.muc.redhat.com/)
- Service:     Check expired date of certificate
- - - - - - - - - - - - - - - - -
- State:       WARNING
- Date:        2026-03-19 10:59:37 (Duration: 0d 0h 0m 0s)
- Output:      WARNING - Certificate '[aap.coe.muc.redhat.com](http://aap.coe.muc.redhat.com/)' expires in 28 day(s) (Fri 17 Apr 2026 09:44:08 AM GMT +0000).

----------------------------------

[sluetze]

The certificate was refreshed using the Ansible Workflow, and the secret was refreshed using oc annotate es aap-coe-cert force-sync=$(date +%s)

however the certificate on the route wasn't updating. On Inspection it seems as if
route_tls_secret: aap-coe-cert is not set in the automationcontroller CR which was contradicted by https://github.com/stormshift/clusters/blob/main/isar-apps/aap/AutomationController.yaml#L10. This configuration is not active, since stormshift/clusters@c52728e .

I manually added route_tls_secret: aap-coe-cert to the ressource and waited for reconciliation, which happened instantly.

Upon inspection of the results, aap.coe was still holding the old cert, but the automation controller wasnt.
obviously in the past there were TWO certificates. One for AAP and one for the automation controller. Utilizing the route_tls_secret in automationcontroller send the automationcontroller into TLS issues, as it has a different URL .

So I requested a new cert

common_name: aap.coe.muc.redhat.com
subject_alt_name: DNS:aap.coe.muc.redhat.com,DNS:aap-controller.apps.isar.coe.muc.redhat.com

This did not work, as the Certificate was not old enough. After enabling "prompt on launch" for extra_vars on the Job Template in AAP I was able to overwrite the check. and finally get a new cert.

oc annotate es aap-coe-cert force-sync=$(date +%s) --overwrite
oc annotate automationcontroller coe-aap redo=$(date +%s) --overwrite
oc annotate ansibleautomationplatform coe redo=$(date +%s) --overwrite

which populated the controller but not the aap interface.

after crying for 5 intense minutes, and looking for alternatives, I just killed the aap-gateway-operator-controller-manager pod to get myself some sweet revenge.

Result LGTM

true | openssl s_client -connect aap.coe.muc.redhat.com:443 -servername aap.coe.muc.redhat.com 2>/dev/null | openssl x509 -dates -ext subjectAltName -noout
notBefore=Mar 26 12:44:05 2026 GMT
notAfter=Mar 21 12:44:05 2027 GMT
X509v3 Subject Alternative Name: 
    DNS:aap.coe.muc.redhat.com, DNS:aap-controller.apps.isar.coe.muc.redhat.com

true | openssl s_client -connect aap-controller.apps.isar.coe.muc.redhat.com:443 -servername aap-controller.apps.isar.coe.muc.redhat.com 2>/dev/null | openssl x509 -noout -dates -ext subjectAltName
notBefore=Mar 26 12:44:05 2026 GMT
notAfter=Mar 21 12:44:05 2027 GMT
X509v3 Subject Alternative Name: 
    DNS:aap.coe.muc.redhat.com, DNS:aap-controller.apps.isar.coe.muc.redhat.com

[rbo]

@sluetze thank you very much!

---------- Forwarded message ---------
From: OMD site mon <mon@monitoring.coe.muc.redhat.com>
Date: Thu, Mar 26, 2026 at 2:14 PM
Subject: *** RECOVERY *** aap.coe.muc.redhat.com / Check expired date of certificate is OK
To: <rbohne@redhat.com>


--SERVICE-ALERT-------------------
-
- Hostaddress: 10.32.98.2
- Hostname:    [aap.coe.muc.redhat.com](http://aap.coe.muc.redhat.com/)
- Service:     Check expired date of certificate
- - - - - - - - - - - - - - - - -
- State:       OK
- Date:        2026-03-26 14:14:38 (Duration: 0d 0h 0m 0s)
- Output:      OK - Certificate '[aap.coe.muc.redhat.com](http://aap.coe.muc.redhat.com/)' will expire on Sun 21 Mar 2027 12:44:05 PM GMT +0000.

----------------------------------