Reference Dataset can get stuck in NotBound forever when a terminating ThinRuntime is treated as existing

[cheyang]

What is your environment?

• Fluid version: master (control plane images v1.1.0-36f0467)
• Kubernetes: v1.36.1 (ACK, 3 nodes)
• Chart: charts/fluid/fluid from master
• Runtime involved: CacheRuntime (curvine) + auto-created ThinRuntime for a reference Dataset

What happened?

A reference Dataset (spec.mounts[0].mountPoint: dataset://<ns>/<name>) stayed in NotBound forever, with no ThinRuntime object in the namespace at all, and no error reported anywhere on the Dataset. The Dataset had a finalizer and a phase, so nothing looked broken — it was simply never reconciled again.

Touching the Dataset (kubectl annotate dataset <ref-dataset> probe=1 --overwrite) recreated the ThinRuntime immediately and the Dataset became Bound within seconds. So the state was wedged, not slow.

This was hit while running test/gha-e2e/curvine/test.sh on a cluster where the same test names had been used by an earlier run.

Timeline (observed, with logs)

1. Leftover from an earlier run: an orphan ThinRuntime/curvine-demo-ref was pending deletion (deletionTimestamp set, finalizer thin-runtime-controller-finalizer still present). Its cleanup was blocked because thinruntime-controller was at replicas: 0 — the chart declares the runtime controllers with replicas: 0 and relies on ScaleoutRuntimeControllerOnDemand to scale them up (a helm upgrade re-applies replicas: 0, which can scale a controller down again while one of its objects is still terminating).

   Warning  ErrorProcessRuntime  thinruntime/curvine-demo-ref
     Process Runtime error Dataset.data.fluid.io "curvine-demo" not found
   

2. New run creates the reference Dataset (14:02:13). dataset-controller scales thinruntime-controller out on demand and adds its finalizer:

   14:02:13.447  utils/runtime_checkers.go:35  Succeed in finding the object  {"runtime": {"kind": "ThinRuntime", "namespace": "default", "name": "curvine-demo-ref"}}
   14:02:13.455  dataset/dataset_controller.go:90   scale out the runtime controller on demand successfully  {"controller": "thinruntime-controller"}
   14:02:13.455  dataset/dataset_controller.go:255  Add finalizer and Requeue  {"dataset": {"name":"curvine-demo-ref"}}
   

   Note Succeed in finding the object — the Get found the terminating leftover, so CreateRuntimeForReferenceDatasetIfNotExist concluded "the runtime already exists" and did not create anything. It also tried to re-parent that dying object by overwriting its ownerReferences.

3. The terminating leftover finally disappears (14:02:39), right after the freshly scaled-out controller starts and removes the finalizer:

   14:02:39.246  leaderelection.go:260  successfully acquired lease fluid-system/thin.data.fluid.io
   14:02:39.663  controllers/runtime_controller.go:233  before clean up finalizer     {"runtime": {"kind": "ThinRuntime", "name": "curvine-demo-ref"}}
   14:02:39.678  controllers/runtime_controller.go:247  Finalizer is removed successfully
   

4. Nothing ever recreates it. The Dataset sits at NotBound indefinitely:

   >>> checking reference dataset.status.phase==Bound (already 555s, last state: NotBound)
   

   $ kubectl get thinruntime -n default
   No resources found in default namespace.
   $ kubectl get dataset -n default
   NAME               PHASE      AGE
   curvine-demo       Bound      10m
   curvine-demo-ref   NotBound   9m10s
   

Root cause

Two independent gaps combine into a permanently wedged state:

A. The auto-create helper treats a terminating runtime as an existing one.

pkg/utils/dataset_runtime.go:45 CreateRuntimeForReferenceDatasetIfNotExist:

runtime, err := GetThinRuntime(client, dataset.GetName(), dataset.GetNamespace())
// 1. if err is null, which indicates that the runtime exists, then return
if err == nil {
    runtimeToUpdate := runtime.DeepCopy()
    runtimeToUpdate.SetOwnerReferences(...)   // re-parents an object that is already dying
    ...
    return nil
}

GetThinRuntime (pkg/utils/runtimes.go:99) is a plain Get, which happily returns objects that have a deletionTimestamp. There is no utils.HasDeletionTimestamp check (the helper exists at pkg/utils/crtl_utils.go:158), so err == nil also covers "exists, but is being deleted and will be gone in a moment".

B. Nothing re-triggers the Dataset once the runtime disappears.

• DatasetReconciler.SetupWithManager (pkg/controllers/v1alpha1/dataset/dataset_controller.go:265) only does For(&datav1alpha1.Dataset{}) — there is no Owns(&ThinRuntime{}) / watch on the runtime it creates, so the deletion of the ThinRuntime produces no reconcile event for the Dataset.
• reconcileDataset ends with return utils.NoRequeue() (dataset_controller.go:181) — the periodic RequeueAfterInterval(r.ResyncPeriod) right above it is only taken when needRequeue is set, and the line that would always requeue is commented out.

So after step 3 there is no event source and no resync that would ever call CreateRuntimeForReferenceDatasetIfNotExist again. The reference Dataset is stuck until a human edits it.

Contributing factor: because the chart ships runtime controllers at replicas: 0 and depends on on-demand scale-out, a helm upgrade (or any re-apply of the chart) can scale a runtime controller back to 0 while one of its objects is terminating. That object then stays in Terminating for an unbounded time with its finalizer intact — which is exactly the leftover that triggers gap A on the next run.

How to reproduce

Deterministic recipe (any runtime works; ThinRuntime for a reference Dataset shown):

1. Create a reference Dataset so a ThinRuntime is auto-created and thinruntime-controller is scaled out.
2. Scale the controller down to simulate the window: kubectl scale deploy thinruntime-controller -n fluid-system --replicas=0 (or run helm upgrade on the chart, which re-applies replicas: 0).
3. Delete the reference Dataset. The ThinRuntime goes to Terminating and stays there (finalizer, no controller).
4. Recreate the reference Dataset with the same name. dataset-controller logs Succeed in finding the object … ThinRuntime and skips creation.
5. The controller gets scaled out again, removes the finalizer, and the object disappears.
6. The Dataset now stays NotBound forever. kubectl annotate dataset <name> probe=1 --overwrite unblocks it.

Impact

• Reference Datasets can wedge permanently with no error surfaced on the Dataset — the only symptom is NotBound and a missing ThinRuntime.
• It is not self-healing: no resync, no watch, no event. Manual intervention is required.
• Most likely to bite CI/e2e (repeated runs reusing the same object names, plus chart re-installs between runs) and any environment where the fluid chart is upgraded while a runtime is being deleted.

Suggested fix

1. In CreateRuntimeForReferenceDatasetIfNotExist, treat a runtime with a deletionTimestamp as not existing yet: if utils.HasDeletionTimestamp(runtime.ObjectMeta), do not adopt/re-parent it, and requeue so the Dataset is retried once the object is really gone. Adopting a terminating object via Update should be avoided in any case.
2. Add Owns(&datav1alpha1.ThinRuntime{}) (or an explicit watch mapping runtime → Dataset) to DatasetReconciler.SetupWithManager, so that deleting the auto-created runtime re-triggers the Dataset and it is recreated.
3. Optionally re-enable a bounded periodic requeue for Datasets that are not yet Bound, as a safety net for any state that no event covers.
4. Separately, consider making the on-demand scale-down refuse to scale a runtime controller to 0 while any of its CRs still have a deletionTimestamp (and/or have the chart not fight on-demand scaling on helm upgrade), so finalizers cannot be stranded.

I can send a PR for 1 + 2 if the direction looks right.

[cheyang]

Fix opened as #6138. What it does, and one correction to the analysis above:

Implemented (suggestions 1 + 2 from the issue):

• CreateRuntimeForReferenceDatasetIfNotExist now reports a ThinRuntime with a deletionTimestamp as an error instead of adopting it and returning success, so the caller requeues until the terminating object is really gone. The error is intentionally not a conflict error so the surrounding retry.RetryOnConflict does not swallow it.
• DatasetReconciler.SetupWithManager now Owns(&datav1alpha1.ThinRuntime{}), so losing the auto-created runtime re-triggers the owning Dataset.

Correction: the issue speculated nothing about it, but while implementing, the first attempt claimed the ownerReference built in CreateRuntimeForReferenceDatasetIfNotExist ends up with empty kind/apiVersion because a typed client clears TypeMeta, and that the API server rejects it. That is not what happens in practice — the ThinRuntime auto-created on the cluster carries a complete ownerReference:

ownerReferences: [{"apiVersion":"data.fluid.io/v1alpha1","controller":true,"kind":"Dataset","name":"curvine-demo-ref","uid":"717ce1a1-..."}]

The PR still adds a fallback to the well-known Dataset kind/apiVersion, since the owner-based watch resolves dependents through exactly those two fields, but it is defensive — not a fix for an observed failure.

Not addressed (still open, tracked by this issue): the contributing factor — the chart ships runtime controllers at replicas: 0 and relies on on-demand scale-out, so a helm upgrade (or any re-apply) can scale a runtime controller back to 0 while one of its objects is terminating, stranding the finalizer. That is what produced the leftover object in the first place. Same latent empty-TypeMeta ownerReference pattern also exists in pkg/controllers/runtime_controller.go and pkg/controllers/operation_controller.go, worth a follow-up.