Add cluster-health-monitor Task and CronJob for dogfooding cluster

[vdemeester]

Changes

Add a lightweight monitoring solution that checks the health of CronJobs,
Jobs, PipelineRuns, and TaskRuns in the dogfooding cluster and creates
GitHub issues when problems are detected.

Problem

We currently have no monitoring or alerting for the Tekton dogfooding
cluster. When things break (image pull failures, node replacements, stuck
jobs), nobody is notified until someone manually checks.

On 2026-02-11, all cluster nodes were replaced. The new nodes run CRI-O which
enforces fully-qualified image names. All 28 cronjobs started failing with
ImagePullBackOff and because they use concurrencyPolicy: Forbid, stuck
jobs blocked all subsequent runs. This went unnoticed for 2+ days.

Solution

A cluster-health-monitor Task + CronJob that runs daily and checks:

1. CronJob health: suspended cronjobs, stuck active jobs blocking
   concurrencyPolicy=Forbid, missed schedules
2. Job health: ImagePullBackOff pods, failed jobs
3. PipelineRun/TaskRun health: recent failures across all monitored
   namespaces (default, tekton-ci, tekton-nightly, bastion-p, bastion-z)

When issues are detected, it creates a GitHub issue in tektoncd/plumbing
with structured labels and a detailed report (with dedup to avoid spam).

Key design decisions

• CronJob → TaskRun directly via kubectl (no EventListener) — independent
  of the trigger infrastructure it monitors
• ghcr.io/ images only (kubectl, tkn) — avoids CRI-O short name issues
  that currently break docker.io/ workloads
• concurrencyPolicy: Replace + activeDeadlineSeconds: 600 — the
  monitor itself can never get stuck
• Shared workspace (emptyDir) passes the report between the 3 steps

Files

tekton/cronjobs/dogfooding/cluster-health-monitor/
├── README.md           # Architecture docs
├── rbac.yaml           # ServiceAccount + ClusterRole/Binding
├── task.yaml           # 3-step Task (check-cronjobs, check-pr-tr, report)
├── cronjob.yaml        # Daily CronJob creating a TaskRun
└── kustomization.yaml  # Kustomize bundle

/kind feature

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

• Includes docs (if user facing)
• Commit messages follow commit message best practices

See the contribution guide
for more details.

[tekton-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please ask for approval from vdemeester after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment