Skip to content

Add a dashboard for Compliance Operator #3304

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
xiaojiey wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add a dashboard for Compliance Operator #3304

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
15c5c57
Add a dashboard for Compliance Operator
xiaojiey Mar 3, 2026
081fbef
Add one more job
xiaojiey Mar 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions config/openshift-customizations.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,73 @@ releases:
periodic-ci-Azure-ARO-HCP-main-periodic-prod-brazilsouth-e2e-parallel: true
periodic-ci-Azure-ARO-HCP-main-periodic-prod-centralindia-e2e-parallel: true
periodic-ci-Azure-ARO-HCP-main-periodic-prod-uksouth-e2e-parallel: true
compliance-operator:
jobs:
# Upstream ComplianceAsCode jobs
periodic-ci-ComplianceAsCode-content-master-4.12-e2e-aws-openshift-node-compliance-arm-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.12-e2e-aws-openshift-node-compliance-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.12-e2e-aws-openshift-platform-compliance-arm-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.12-e2e-aws-openshift-platform-compliance-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.14-e2e-aws-openshift-node-compliance-arm-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.14-e2e-aws-openshift-platform-compliance-arm-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.16-e2e-aws-openshift-node-compliance-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.16-e2e-aws-openshift-platform-compliance-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.17-e2e-aws-openshift-node-compliance-arm-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.17-e2e-aws-openshift-platform-compliance-arm-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.18-e2e-aws-openshift-node-compliance-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.18-e2e-aws-openshift-platform-compliance-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.19-e2e-aws-openshift-node-compliance-arm-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.19-e2e-aws-openshift-platform-compliance-arm-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.20-e2e-aws-openshift-node-compliance-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.20-e2e-aws-openshift-platform-compliance-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.21-e2e-aws-openshift-node-compliance-arm-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.21-e2e-aws-openshift-node-compliance-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.21-e2e-aws-openshift-platform-compliance-arm-weekly: true
periodic-ci-ComplianceAsCode-content-master-4.21-e2e-aws-openshift-platform-compliance-weekly: true
# Downstream compliance operator jobs
periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-aws-ipi-proxy-fips-f60-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-aws-ipi-proxy-fips-f60-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-4.13-amd64-nightly-azure-ipi-fullyprivate-proxy-f60-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.13-amd64-nightly-azure-ipi-fullyprivate-proxy-f60-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-nightly-gcp-ipi-proxy-etcd-encryption-f60-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.14-arm64-nightly-azure-ipi-private-f60-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.14-arm64-nightly-azure-ipi-private-f60-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-nightly-vsphere-ipi-ovn-dualstack-privmaryv6-f60-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-nightly-vsphere-ipi-ovn-dualstack-privmaryv6-f60-co-destructive: true
periodic-ci-openshift-openshift-tests-private-release-4.15-arm64-nightly-baremetal-upi-ovn-ipv4-f60-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.15-arm64-nightly-baremetal-upi-ovn-ipv4-f60-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-nightly-azure-stack-ipi-proxy-fips-f28-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-nightly-azure-stack-ipi-proxy-fips-f28-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-4.16-arm64-nightly-gcp-ipi-f28-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.16-arm64-nightly-gcp-ipi-f28-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-4.16-multi-nightly-gcp-ipi-ovn-ipsec-amd-mixarch-f28-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-aws-ipi-proxy-sts-f28-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-aws-ipi-proxy-sts-f28-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-4.17-arm64-nightly-aws-ipi-disc-priv-sts-ep-f28-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.17-arm64-nightly-aws-ipi-disc-priv-sts-ep-f28-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-4.18-amd64-nightly-gcp-ipi-ovn-winc-f14-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.18-amd64-nightly-gcp-ipi-ovn-winc-f14-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-4.18-arm64-nightly-azure-ipi-f14-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.18-arm64-nightly-azure-ipi-f14-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-4.18-multi-nightly-aws-ipi-ovn-ipsec-arm-mixarch-f14-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.19-amd64-nightly-aws-ipi-ovn-hypershift-fips-guest-f999-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.19-amd64-nightly-baremetalds-ipi-ovn-lvms-f14-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.19-amd64-nightly-baremetalds-ipi-ovn-lvms-f14-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-4.19-arm64-nightly-gcp-ipi-proxy-private-f28-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.19-arm64-nightly-gcp-ipi-proxy-private-f28-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-4.20-amd64-nightly-baremetalds-ipi-ovn-lvms-f14-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.20-amd64-nightly-baremetalds-ipi-ovn-lvms-f14-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-4.20-arm64-nightly-gcp-ipi-proxy-private-f28-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.20-arm64-nightly-gcp-ipi-proxy-private-f28-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-4.21-amd64-nightly-baremetalds-ipi-ovn-lvms-f14-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.21-amd64-nightly-baremetalds-ipi-ovn-lvms-f14-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-4.21-arm64-nightly-gcp-ipi-proxy-private-f28-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.21-arm64-nightly-gcp-ipi-proxy-private-f28-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-baremetalds-ipi-ovn-lvms-f14-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-baremetalds-ipi-ovn-lvms-f14-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-4.22-arm64-nightly-gcp-ipi-proxy-private-f28-compliance: true
periodic-ci-openshift-openshift-tests-private-release-4.22-arm64-nightly-gcp-ipi-proxy-private-f28-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-baremetalds-ipi-ovn-lvms-f14-compliance: true
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-baremetalds-ipi-ovn-lvms-f14-compliance-destructive: true
periodic-ci-openshift-openshift-tests-private-release-5.0-arm64-nightly-gcp-ipi-proxy-private-f28-compliance: true
periodic-ci-openshift-openshift-tests-private-release-5.0-arm64-nightly-gcp-ipi-proxy-private-f28-compliance-destructive: true
Comment on lines +35 to +104
Copy link
Contributor

@coderabbitai coderabbitai bot Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# 1) First, extract and display the actual compliance-operator jobs from the file
echo "=== Extracting jobs from config/openshift-customizations.yaml ==="
jobs=$(awk '
  /^  compliance-operator:/ {in_block=1; next}
  in_block && /^[^ ]/ {in_block=0}
  in_block && /^      periodic-ci-/ {
    line=$0
    sub(/: true[[:space:]]*$/, "", line)
    gsub(/^[[:space:]]+/, "", line)
    print line
  }
' config/openshift-customizations.yaml)

echo "Found jobs:"
echo "$jobs" | nl
echo ""

# 2) Check for the suspected typo
echo "=== Checking for 'privmaryv6' typo ==="
if echo "$jobs" | grep -i "privmaryv6"; then
  echo "FOUND: privmaryv6 typo detected"
else
  echo "No 'privmaryv6' found"
fi
echo ""

# 3) Query Prow prowjobs.js
echo "=== Fetching Prow jobs catalog ==="
curl -fsSL "https://prow.ci.openshift.org/prowjobs.js" -o /tmp/prowjobs.js 2>&1 | head -5

if [ -f /tmp/prowjobs.js ]; then
  echo "Prow catalog fetched successfully"
  
  # 4) Verify each job exists in Prow
  echo ""
  echo "=== Verifying job existence in Prow ==="
  missing=0
  found=0
  while IFS= read -r j; do
    if grep -q "\"name\":\"$j\"" /tmp/prowjobs.js; then
      found=$((found + 1))
    else
      echo "MISSING: $j"
      missing=$((missing + 1))
    fi
  done <<< "$jobs"
  
  echo ""
  echo "Summary: Found=$found, Missing=$missing"
else
  echo "ERROR: Could not fetch Prow jobs catalog"
fi

Repository: openshift/sippy

Length of output: 197

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# 1) Check file exists and show the actual compliance-operator section
echo "=== Checking file and compliance-operator section ==="
if [ ! -f config/openshift-customizations.yaml ]; then
  echo "ERROR: config/openshift-customizations.yaml not found"
  exit 1
fi

# Show lines around compliance-operator to understand structure
echo "File content around compliance-operator:"
grep -n "compliance-operator" config/openshift-customizations.yaml | head -5
echo ""

# 2) Extract jobs using simpler grep + sed approach
echo "=== Extracting jobs ==="
jobs=$(grep "periodic-ci-" config/openshift-customizations.yaml | sed 's/.*periodic-ci-/periodic-ci-/' | sed 's/: true.*//')
echo "Total jobs extracted:"
echo "$jobs" | wc -l
echo ""

# 3) Show first few jobs
echo "First 10 jobs:"
echo "$jobs" | head -10
echo ""

# 4) Check for suspected typo
echo "=== Checking for 'privmaryv6' typo ==="
if echo "$jobs" | grep -i "privmaryv6"; then
  echo "FOUND: privmaryv6 typo detected"
else
  echo "No 'privmaryv6' found in job names"
fi
echo ""

# 5) Fetch Prow catalog
echo "=== Fetching Prow jobs catalog ==="
timeout 30 curl -fsSL "https://prow.ci.openshift.org/prowjobs.js" -o /tmp/prowjobs.js || {
  echo "Warning: Could not fetch from prow.ci.openshift.org, trying alternative source"
  timeout 30 curl -fsSL "https://prow.ci.openshift.org/api/prowjobs" -o /tmp/prowjobs.json || true
}

if [ -f /tmp/prowjobs.js ]; then
  echo "Prow catalog (prowjobs.js) fetched, size: $(wc -c < /tmp/prowjobs.js) bytes"
  echo "Sample of catalog:"
  head -c 500 /tmp/prowjobs.js
  echo ""
elif [ -f /tmp/prowjobs.json ]; then
  echo "Prow catalog (API JSON) fetched, size: $(wc -c < /tmp/prowjobs.json) bytes"
else
  echo "ERROR: Could not fetch Prow catalog from either endpoint"
fi

Repository: openshift/sippy

Length of output: 1373

🌐 Web query:

openshift vsphere dualstack ipv6 periodic test job name compliance

💡 Result:

In OpenShift CI/Prow, periodic job names are expected to follow the standard pattern:

periodic-ci-<org>-<repo>-<branch>-<test-context>

You can see this pattern in existing vSphere periodics like periodic-ci-openshift-release-master-nightly-4.14-e2e-vsphere-ovn-csi / ...-e2e-vsphere-ovn-serial / ...-e2e-vsphere-ovn-upi. [1]

For the vSphere OVN dual-stack lane, there is already a corresponding PR job named:

pull-ci-openshift-cluster-network-operator-master-e2e-vsphere-ovn-dualstack [2]

So, a compliant periodic name for the same lane would be:

periodic-ci-openshift-cluster-network-operator-master-e2e-vsphere-ovn-dualstack [2]

If you specifically need to distinguish IPv6-only vs dual-stack, OpenShift CI convention is to encode that in the trailing context (e.g., ...-ipv6 vs ...-dualstack), similar to other existing periodics (for non-vSphere platforms) that use ...-ovn-ipv6 and ...-ovn-dualstack. [3]

Sources:
[1] [4.14 nightly payload test list showing vSphere periodic job names] (amd64.ocp.releases.ci.openshift.org)
[2] [Existing vSphere OVN dual-stack CI job name (PR job)] (prow.ci.openshift.org)
[3] [Example periodic naming for ipv6 vs dualstack in payload tests] (openshift-release.apps.ci.l2s4.p1.openshiftapps.com)

Citations:

Fix malformed job name on line 66: privmaryv6 is a typo and will cause silent data loss.

Line 66 contains periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-nightly-vsphere-ipi-ovn-dualstack-privmaryv6-f60-compliance, where dualstack-privmaryv6 violates OpenShift CI naming conventions. The privmaryv6 token is malformed (likely a typo for primaryv6 or should be removed entirely). This job will not resolve in Prow, and the compliance dashboard will silently show no data for the 4.15 vSphere dual-stack compliance variant.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@config/openshift-customizations.yaml` around lines 35 - 103, The job name
string
periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-nightly-vsphere-ipi-ovn-dualstack-privmaryv6-f60-compliance
contains a typo ("privmaryv6") that breaks CI naming; update that token to
"primaryv6" (resulting name:
periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-nightly-vsphere-ipi-ovn-dualstack-primaryv6-f60-compliance)
so the entry conforms to OpenShift CI naming and will resolve correctly in Prow.