Update on-call log with incidents for 2025-11-17

Added on-call log entries for the week of 2025-11-17, detailing incidents and actions taken.

Author: pieterbeulque

engineering/on-call-log.mdx

@@ -10,6 +10,23 @@ When everything is running smoothly, the on-call engineer can handle maintenance
 
 The goal of this rolling log is to ease handover between on-call for unresolved issues, and keep a log of what's been handled recently.
 
+## Week of 2025-11-17
+
+On-call: Pieter Beulque
+
+### Incidents
+
+#### 2025-11-22
+
+We had a jump scare on Saturday because of the undelivered webhooks monitor, but it was a false alarm, the queue grew up to 17 undelivered webhooks but gradually backed down again.
+I'm not sure there was an actual root cause or just a traffic spike. We can reconsider the threshold for the monitor because it was arbitrarily set to 10 back in the day. 
+
+#### 2025-11-17
+
+Our worker went out of memory because of a huge spike in subscription renewals. We have a merchant that is using daily subscriptions for a use case. This in turn triggered a lot of invoices trying to generate concurrently and the invoice number unique index caused the database to lock.
+
+As a mitigation we moved the merchant to the new customer-based invoice indexing. As a future mitigation we also made this default for new organizations. The same issue may still happen for existing organizations that see sudden spikes in orders but the same fix can then also be applied. 
+
 ## Week of 2025-11-10
 
 On-call: Jesper Bränn