Skip to content

Idea: Streamline scaffolded controller and e2e tests with Sawchain #7085

@eolatham

Description

@eolatham

Context

A fresh operator-sdk init + create api project scaffolds two test surfaces:

  • internal/controller/<kind>_controller_test.go — a Ginkgo skeleton on envtest + the controller-runtime client, stopping at a single Reconcile call with a // TODO(user): Add more specific assertions comment.
  • test/e2e/e2e_test.go — a Ginkgo suite driving everything through exec.Command("kubectl", ...) with stdout parsing.

Sawchain is a Go library (built on Chainsaw) with Gomega integration that works equally well for both: low-level integration tests against envtest and full e2e tests against a live cluster, using the same API on top of the same controller-runtime client. It offers:

  • Readable, declarative assertions — YAML describes the expected resource state instead of repetitive/imperative Get + field-by-field checks.
  • Partial matching with JMESPath — assert only the fields you care about, and bind values across assertions.
  • Ergonomic lifecycle helpersCreateAndWait, UpdateAndWait, DeleteAndWait bundle each write with an observability wait, so tests don't race against client-cache sync.
  • Mix-and-match — YAML-driven and struct-based logic coexist in the same test; pick whichever fits each assertion.

I wanted to share it as something that could simplify and enhance both scaffolded test surfaces.

Controller integration tests (internal/controller/...)

Any realistic test past the scaffolded TODO quickly grows into repetitive client.Get + field-by-field assertion loops.

Traditional pattern (simplified from a PodSet controller example):

podSet := &v1.PodSet{
    ObjectMeta: metav1.ObjectMeta{Name: "test-podset", Namespace: "default"},
    Spec: v1.PodSetSpec{
        Replicas: ptr.To(2),
        Template: v1.Template{
            Name:       "test-pod",
            Containers: []v1.Container{{Name: "test-app", Image: "test/app:v1"}},
        },
    },
}
Expect(k8sClient.Create(ctx, podSet)).To(Succeed())

Eventually(func() error {
    if err := k8sClient.Get(ctx,
        client.ObjectKeyFromObject(podSet), podSet); err != nil {
        return err
    }
    if len(podSet.Status.Pods) != 2 {
        return fmt.Errorf("expected 2 pods, got %d", len(podSet.Status.Pods))
    }
    return nil
}).Should(Succeed())

for _, podName := range podSet.Status.Pods {
    pod := &corev1.Pod{}
    Expect(k8sClient.Get(ctx, client.ObjectKey{Name: podName, Namespace: "default"}, pod)).To(Succeed())
    Expect(pod.Spec.Containers).To(HaveLen(1))
    Expect(pod.Spec.Containers[0].Name).To(Equal("test-app"))
    Expect(pod.Spec.Containers[0].Image).To(Equal("test/app:v1"))
}

With Sawchain:

podSet := &v1.PodSet{}
sc.CreateAndWait(ctx, podSet, `
    apiVersion: apps.example.com/v1
    kind: PodSet
    metadata:
      name: test-podset
      namespace: ($namespace)
    spec:
      replicas: 2
      template:
        name: test-pod
        containers:
        - name: test-app
          image: test/app:v1
`)

Eventually(sc.FetchSingleFunc(ctx, podSet)).Should(HaveField("Status.Pods", ConsistOf(
    "test-pod-0",
    "test-pod-1",
)))

for _, podName := range podSet.Status.Pods {
    Eventually(sc.CheckFunc(ctx, `
        apiVersion: v1
        kind: Pod
        metadata:
          name: ($name)
          namespace: ($namespace)
        spec:
          containers:
          - name: test-app
            image: test/app:v1
    `, map[string]any{"name": podName})).Should(Succeed())
}

CreateAndWait handles lifecycle, CheckFunc expresses declarative assertions with partial matching and JMESPath, and FetchSingleFunc polls state. There are also helpers like HaveStatusCondition, MatchYAML, and many more — check out the design overview for the full capabilities.

E2E tests (test/e2e/...)

The scaffolded e2e suite verifies cluster state by shelling out to kubectl and parsing the output: fragile, deeply coupled to kubectl's output shape, and hard to extend.

Scaffolded pattern (from e2e_test.go):

verifyCurlUp := func(g Gomega) {
    cmd := exec.Command("kubectl", "get", "pods", "curl-metrics",
        "-o", "jsonpath={.status.phase}",
        "-n", namespace)
    output, err := utils.Run(cmd)
    g.Expect(err).NotTo(HaveOccurred())
    g.Expect(output).To(Equal("Succeeded"), "curl pod in wrong status")
}
Eventually(verifyCurlUp, 5*time.Minute).Should(Succeed())

With Sawchain (same controller-runtime client, pointed at the live cluster):

Eventually(sc.CheckFunc(ctx, `
    apiVersion: v1
    kind: Pod
    metadata:
      name: curl-metrics
      namespace: ($namespace)
    status:
      phase: Succeeded
`, map[string]any{"namespace": namespace})).Should(Succeed())

The same pattern replaces the rest of the scaffold's kubectl-shell-out sprawl: namespace setup, pod polling, endpoint checks, token minting. No exec.Command, no stdout parsing, no jsonpath templates.

More

I would love to hear whether this kind of integration could be valuable. Happy to discuss further! 😄

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions