[Bug] Frontend/Admin SearchAttributes and AddOrUpdateRemoteCluster handlers leak an uncached grpc.ClientConn per call

[tz-torchai]

Expected Behavior

Calling {Add,Remove,List/Get}SearchAttributes or AddOrUpdateRemoteCluster repeatedly against temporal-frontend should not cause unbounded goroutine or memory growth — connections used internally to serve these RPCs should be reused/cached or explicitly closed.

Actual Behavior

OperatorHandlerImpl and the legacy AdminHandler each dial a fresh grpc.ClientConn on every call to 4 RPCs (8 call sites total, listed below) and never close it. Neither client-factory method backing these calls caches the connection, so goroutines (grpcsync.(*CallbackSerializer).run, dns.(*dnsResolver).watcher, transport.(*http2Server).keepalive, etc.) and memory grow roughly linearly with call count and are never reclaimed
short of restarting the process. Possibly the underlying cause of goroutine-leak reports like #6323, which never pinpoints a specific call path.

Call sites (confirmed on main HEAD, also present in v1.31.2):

• service/frontend/operator_handler.go
  • addSearchAttributesSQL (:253)
  • removeSearchAttributesSQL (:423)
  • listSearchAttributesSQL (:510)
  • AddOrUpdateRemoteCluster (:610)
• service/frontend/admin_handler.go
  • addSearchAttributesSQL (:374)
  • removeSearchAttributesSQL (:513)
  • getSearchAttributesSQL (:620)
  • AddOrUpdateRemoteCluster (:1203)

Mechanism:

• The 6 SearchAttributes sites call clientFactory.NewLocalFrontendClientWithTimeout(...). Its first return value is the grpc.ClientConnInterface — every call site discards it (_, client, err := ...) and never calls .Close().
• The 2 AddOrUpdateRemoteCluster sites call clientFactory.NewRemoteAdminClientWithTimeout(...), which only returns the RPC client, not the connection at all — the caller has no way to close it even if it wanted to.
• Neither factory method caches the connection: NewLocalFrontendClientWithTimeout calls RPCFactory.CreateLocalFrontendGRPCConnection() (common/rpc/rpc.go:223), which dials fresh via the internal dial() helper on every invocation. Its sibling CreateLocalFrontendHTTPClient() (rpc.go:282) is already sync.OnceValue-cached (rpc.go:88) — this looks like an inconsistency rather than a deliberate per-call design.
• The 6 SearchAttributes sites are gated to SQL-based visibility (skipped on Elasticsearch); the 2 AddOrUpdateRemoteCluster sites are unconditional.

Suggested fix: cache the connection the same way CreateLocalFrontendHTTPClient already does (e.g. sync.OnceValue, keyed appropriately for AddOrUpdateRemoteCluster's per-address case), or at minimum return the connection to callers so they can defer Close().

Steps to Reproduce the Problem

1. Start a temporal-frontend with SQL-based visibility (e.g. Postgres/MySQL, not Elasticsearch).
2. Repeatedly call any of the 8 RPCs above (e.g. via tctl/temporal CLI or a raw gRPC client) — a few hundred calls is enough to see the trend clearly.
3. Compare num_goroutines on the frontend's metrics endpoint before/after, or diff two /debug/pprof/goroutine?debug=1 dumps — goroutine count climbs roughly 1:1 with call count and never drops.

Specifications

• Version: v1.31.2 (call sites also confirmed present, same line numbers, on current main HEAD)
• Platform: N/A — server-side Go code, not platform-specific