SessionProxy never reconnects after retries are exhausted — a transient DB outage disables the archive and semaphore locking until restart

[zelig81]

Pre-requisites

• I have double-checked my configuration
• I have searched existing issues (no match for session proxy is closed)
• I am using the latest release at time of filing

Version

v4.1.1

What happened

A managed-PostgreSQL maintenance restart made the database unreachable for ~10 minutes. Both argo-server and workflow-controller exhausted their reconnect retries during that window and never recovered, even after the database had been healthy for hours. Every subsequent DB operation returned session proxy is closed. Only restarting the processes fixed it.

Cause

In util/sqldb/session.go, reconnectLocked() sets sp.closed = true before each connect attempt, and only a successful connect() clears it:

for attempt := 0; attempt <= sp.maxRetries; attempt++ {
    if sp.sess != nil {
        sp.sess.Close()
        sp.closed = true      // set before the attempt
    }
    err = sp.connect(ctx)     // clears sp.closed only on success
    if err == nil { return nil }
    ...
}
return fmt.Errorf("reconnection failed after %d retries, last error: %w", sp.maxRetries, err)

When every attempt fails, closed stays true permanently. With() then short-circuits on it and never attempts another reconnect:

sp.mu.RLock()
if sp.closed {
    sp.mu.RUnlock()
    logger.Warn(ctx, "session proxy is closed")
    return fmt.Errorf("session proxy is closed")
}

So an outage shorter than the retry budget is survived, and one longer than it is permanent.

Impact

argo-server and workflow-controller hold independent sessions and can wedge separately.

• Server — the workflow archive is unreachable. The list endpoint merges live and archived results, so the UI returns {"code":13,"message":"session proxy is closed"} instead of the workflow list.
• Controller — every semaphore/mutex operation fails. Workflows fail immediately with Failed to acquire the synchronization lock. session proxy is closed, and the lock_manager component logs the error continuously (we saw >15k lines in one 3-hour window).

Nothing detects it. The processes stay healthy: pods remain Ready, the leader lease keeps renewing, and the server's readiness probe targets /, which serves the static UI and returns 200 regardless of database state. One of our clusters ran in this state for ~8 hours before a human noticed a failed workflow.

What you expected to happen

After the database becomes reachable again, subsequent operations should attempt to re-establish the session and recover without an operator restarting the process.

How to reproduce it

1. Configure persistence (archive) and/or synchronization.postgresql (DB semaphores).
2. Make the database unreachable for longer than maxRetries × backoff — a managed-database maintenance restart or failover does this naturally.
3. Restore the database.
4. Every archive read / lock operation continues to fail with session proxy is closed indefinitely. Restarting the pod is the only fix.

Suggested fix

Do not leave closed = true on a failed reconnect — either reset it when the attempts are exhausted so a later With() can retry, or have With() attempt a reconnect when it finds the proxy closed rather than returning immediately.

A related second problem (semaphore rows leaked by the wedged controller, with no recovery path) is filed separately.

Make it detectable, not just recoverable

Even with the reconnect fixed, a wedged session is currently invisible from the outside, which is why ours went unnoticed for hours. Two things would help independently of the fix:

1. Reflect session health in the readiness/liveness endpoints, so Kubernetes can restart a wedged process by itself. Today the server's readiness probe targets /, which serves the static UI and returns 200 regardless of database state — it cannot fail on this. There is no health endpoint that touches the session at all: to get a probe that does, we had to point one at /api/v1/archived-workflows?listOptions.limit=1, which is the only route we found that reaches the session unconditionally and cheaply.

2. Expose it as a metric — for example a gauge for session open/closed, or a counter of exhausted reconnects — so it can be alerted on rather than discovered by a user reporting a broken UI or a failed workflow.

I could not find either today, though I would be glad to be wrong:

• argo-server's /metrics exposes only grpc_server_* and Go runtime series — nothing about the DB session.
• The controller's argo_workflows_error_count carries causes CronWorkflowSpecError, CronWorkflowSubmissionError and OperationPanic. None covers a broken DB session, and none incremented during our outage.

If such a signal already exists and I simply missed it, documenting it would be just as valuable as adding one — the failure is silent by default, so operators need to be told what to watch.