Keeper: watch.namespaces configuration ignored for ClickHouseKeeperInstallation resources

[ligiamonne]

Description

The clickhouse-operator is ignoring the watch.namespaces configuration for ClickHouseKeeperInstallation (CHK) resources. The operator correctly respects namespace watching for ClickHouseInstallation (CHI) resources, but continues to reconcile CHK resources in namespaces outside the configured watch list.

This breaks multi-operator deployments where multiple operators run in the same cluster, each watching different sets of namespaces. Both operators are trying to reconcile CHK resources, causing crashes and inconsistent states.

Environment

• Helm chart altinity-clickhouse-operator

We have 2 operators running in the same cluster (only 1 operator installs CRDs, others skips them), each configured to watch different namespaces:

Operator 1: Does not use Keeper

• Version: 0.25.5
• Configured to watch:

configs:
  files:
    config.yaml:
      watch:
        namespaces:
          - a
          - b
          - c

Operator 2: Uses Keeper

- Version: 0.25.6
- Configured to watch:
configs:
  files:
    config.yaml:
      watch:
        namespaces:
          - d
          - f

Behavior

Expected

Each operator should only manage resources in their configured namespaces.

Actual

Operator 1 is managing CHK resources configured outside its watch list:

1. Correctly loads its namespace configuration at startup
2. Respects namespace boundaries for CHI resources
3. Ignores namespace configuration for CHK resources and reconciles them cluster-wide despite not being configured to watch those namespaces

Operator 2:
Correctly manages the same CHK resources (as configured)

Result

CHK resources are dual-managed by both operators, causing reconciliation conflicts.

Evidence

Operator 1 startup logs (correct configuration loading)

runtime:
    configFilePath: /etc/clickhouse-operator/config.yaml
watch:
    namespaces:
        include:
            - a
            - b
            - c
        exclude: null

15 Seconds Later: Operator 1 reconciling non-watched namespace

I0213 09:18:33.056397       1 worker-pdb.go:36] PDB updated:
d/chk-clickhouse-keeper-single
I0213 09:18:33.057675       1 statefulset-reconciler.go:103] unknown:Have
StatefulSet available, try to perform label-based comparison for sts:
d/chk-clickhouse-keeper-single-0-0
I0213 09:18:33.173256       1 worker-reconciler-chk.go:341]
worker-reconciler-chk.go:341:reconcileHostStatefulSet():start:Host:0-0[0/0]:d/clickhouse-keeper:reconcile StatefulSet start

Namespace d is not in the global operator's configured watch list, yet it is actively reconciling CHK resources in that namespace.

CHK Resource status shows wrong operator:

Both CHK resources in d and f show chop-ip IP of operator 1. This is incorrect, they should
show IP of operator 2. CHI resources in the same namespaces correctly show operator 2 IP, suggests namespace isolation works for CHI but not for CHK.

[sunsingerus]

Thanks for the detailed report and the clear evidence. The bug is confirmed — we'll take a look.

The short version of what's happening: the CHK controller uses controller-runtime, and its cache is scoped to all namespaces when more than one namespace is configured (a known limitation of GetInformerNamespace()). The namespace predicate in keeperPredicate() correctly filters direct CHK events, but it doesn't cover reconcile requests triggered by owned StatefulSet changes — those bypass the predicate and land in Reconcile() with no namespace guard, which is where the unwanted reconciliation comes from.

[alex-zaitsev]

@ligiamonne , should be fixed in https://github.com/Altinity/clickhouse-operator/releases/tag/release-0.26.2, please confirm