fix(backend): report CREATE DATABASE retry failures during DB init

[hsinhoyeh]

Fixes item 7 of #13956 — the DB initialisation failure that produces no log output at all.

initDBDriver retries CREATE DATABASE with backoff.Retry, which discards the error from every attempt:

https://github.com/kubeflow/pipelines/blob/master/backend/src/apiserver/client_manager/client_manager.go#L441-L445

Nothing is logged for the whole InitConnectionTimeout window (6m by default), and the startup probe restarts the container long before that deadline lets util.TerminateIfError turn the failure into a fatal. The error therefore never surfaces — not as a warning, not as a fatal, not at all.

Symptom

An api-server that logs three lines and then appears to hang forever:

I client_manager.go:275] Initializing client manager
I client_manager.go:276] Initializing DB client...
I config.go:75] Config DBConfig.MySQLConfig.ExtraParams not specified, skipping

Every misconfiguration reached through this path looks identical from the outside — an unresolvable database host, a wrong password, a driver pointed at the wrong engine — and identical to a slow but healthy start.

This is what made the other items in #13956 take hours to diagnose rather than seconds. It is the smallest change in that issue and, I would argue, the one with the best diagnostic return.

Change

Use backoff.RetryNotify, matching the sql.Open retry immediately above it (client_manager.go:425), and include the database name and the next retry interval so a failing attempt names its own cause:

E client_manager.go:448] Failed to create database mlpipeline: dial tcp: lookup postgres: no such host. Retrying in 1.5s

This is logging only. The retry policy, the backoff parameters, and the eventual util.TerminateIfError are all unchanged — no behavioural difference beyond the log line.

Testing

initDBDriver opens real database connections and terminates the process on failure, so it has no unit tests today and this PR does not add any; the change is a logging call inside that function.

Verified locally:

• go build ./src/apiserver/... — passes
• go vet ./src/apiserver/client_manager/ — passes
• go test ./src/apiserver/client_manager/ — ok ... 7.251s
• gofmt — clean

Note for anyone reproducing: the storage package needs CGO_ENABLED=1, since SQLiteDialect.IsDuplicateError uses mattn/go-sqlite3, which only exposes sqlite3.Error and sqlite3.ErrConstraint under cgo.

Scope

This is deliberately one item from #13956, kept separate so it can be reviewed on its own. The manifest items (3–6) are in #13958. The storage-layer items (1 and 2) need a design decision and are discussed in #13956 (comment).

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chensun for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• backend/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[google-oss-prow]

Hi @hsinhoyeh. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.