chore: migrate from kuttl to chainsaw

[eddycharly]

Chainsaw was created because kuttl stopped to be actively maintained (it's actually a single maintainer now).

We are also solving most of the kuttl limitations that exist today and wrote a command to automatically migrate from kuttl to chainsaw. It supercharges e2e tests with better logs, config maps assertions, assertions trees and many more things 💪

This is a relatively young project but it has already been adopted by a couple of e2e tests heavy projects (kyverno, keptn, fairwinds rbac-manager, grafana-operator, redis-operator and others).

To give more context, you can see similar PRs at:

• chore: replace kuttl tests with chainsaw keptn/lifecycle-toolkit#3000
• chore: migrate from kuttl to chainsaw open-telemetry/opentelemetry-operator#2630
• chore: migrate from kuttl to chainsaw grafana/grafana-operator#1406
• test(e2e): replace kuttl with chainsaw statnett/image-scanner-operator#794
• feat: add chainsaw scorecard test support operator-framework/operator-sdk#6676
• chore: migrate from kuttl to chainsaw grafana/tempo-operator#786 in this PR the tempo-operator maintainers asked some good questions and I explained the current status, that I suggest taking a look at.

Sadly, kuttl haven't done a release since jan 2023.

Here you can find a good video showcasing chainsaw: https://www.youtube.com/watch?v=Ejof-wtAdQM&ab_channel=CamilaMacedo

To make things more concrete, there are a couple of issues in this repo, a few files are not valid and kuttl doesn't complain because unmarshalling is not strict so the non-valid fields are simply ignored. This is a problem because the behaviour of the tests is not the expected one, for example:

• argocd-operator/tests/k8s/1-006_validate-managed-by-chain/09-delete-app-and-namespace.yaml

  Lines 14 to 15 in 4bd23cb

  	command:
  	- shell: sleep 5

• argocd-operator/tests/olm/1-001_alpha_to_beta_dex_conversion/02-delete.yaml

  Lines 1 to 7 in 4bd23cb

  	apiVersion: kuttl.dev/v1beta1
  	kind: TestStep
  	delete:
  	- apiVersion: argoproj.io/v1alpha1
  	kind: ArgoCD
  	metadata:
  	name: argocd

• argocd-operator/tests/olm/1-003_alpha_to_beta_sso_conflict_conversion/02-delete.yaml

  Lines 1 to 7 in 4bd23cb

  	apiVersion: kuttl.dev/v1beta1
  	kind: TestStep
  	delete:
  	- apiVersion: argoproj.io/v1alpha1
  	kind: ArgoCD
  	metadata:
  	name: argocd

• argocd-operator/tests/olm/1-002_alpha_to_beta_keycloak_conversion/02-delete.yaml

  Lines 1 to 9 in 4bd23cb

  	apiVersion: kuttl.dev/v1beta1
  	kind: TestStep
  	delete:
  	- apiVersion: argoproj.io/v1alpha1
  	kind: ArgoCD
  	metadata:
  	name: argocd
  	labels:
  	example: keycloak

If you need more infos 👇

• GitHub: https://github.com/kyverno/chainsaw
• Docs: https://kyverno.github.io/chainsaw
• Slack: https://kubernetes.slack.com/archives/C067LUFL43U

[iam-veeramalla]

@eddycharly Really appreciate the contribution. We will go through the PR. Thanks again :)

[eddycharly]

@iam-veeramalla once everything is green, i will see how to enable concurrent tests. It should be easy, chainsaw has a feature to disable concurrency on a per test basis.

[eddycharly]

@iam-veeramalla any chance to get this in ?

[eddycharly]

i bumped chainsaw to the latest version