Diagnostic TS removal (#591)

Non of the existing TCs in the diagnostic test suite is actually a TC,
since they cannot fail. They're just retrieving information from the
cluster, so the only way for them to fail comes from network or cluster
permissions related errors. On those cases, the output of any other
TS/TC would also be unreliable/unstrustable.

This PR makes the diagnostic retrieval to be done right before any other
TS, so the claim.json will always have that information, no matter which
TS was launched.

The flag.Parse() function was not needed, since ginkgo calls it
internally before calling TestTest() function. This second flag.Parse()
call was reparsing everything and creating duplicates in the ginkgo
config slices like FocusStrings/SkipStrings.

+ Some refactors, plus discarding -s and -l flags when -f is not set.

Author: greyerof

.github/workflows/pre-main.yaml

@@ -157,7 +157,7 @@ jobs:
       # Perform smoke tests.
 
       - name: 'Test: Run test suites'
-        run: ./run-cnf-suites.sh --focus access-control diagnostic lifecycle platform observability networking affiliated-certification operator
+        run: ./run-cnf-suites.sh --focus access-control lifecycle platform observability networking affiliated-certification operator
 
       - name: Upload smoke test results as an artifact
         uses: actions/upload-artifact@v2
@@ -188,8 +188,8 @@ jobs:
           cp test-network-function/*.yml $TNF_CONFIG_DIR
         shell: bash
 
-      - name: 'Test: Run diagnostic test suite in a TNF container'
-        run: ./run-tnf-container.sh ${{ env.TESTING_CMD_PARAMS }} -f diagnostic
+      - name: 'Test: Run without any TS, just get diagnostic information'
+        run: ./run-tnf-container.sh ${{ env.TESTING_CMD_PARAMS }}
 
       - name: Delete the deployment and deploy Statefulset pods
         uses: ./cnf-certification-test-partner/.github/actions/deploy-statefulset-pods

.github/workflows/tnf-image.yaml

@@ -141,8 +141,8 @@ jobs:
           cp test-network-function/*.yml $TNF_CONFIG_DIR
         shell: bash
 
-      - name: 'Test: Run diagnostic test suite in a TNF container'
-        run: ./run-tnf-container.sh ${{ env.TESTING_CMD_PARAMS }} -f diagnostic      
+      - name: 'Test: Run without any TS, just get diagnostic information'
+        run: ./run-tnf-container.sh ${{ env.TESTING_CMD_PARAMS }} 
 
       # Push the new TNF image to Quay.io.
 

CATALOG.md

@@ -111,69 +111,6 @@ Result Type|normative
 Suggested Remediation|Ensure that your Operator has passed Red Hat's Operator Certification Program (OCP).
 Best Practice Reference|[CNF Best Practice V1.2](https://connect.redhat.com/sites/default/files/2021-03/Cloud%20Native%20Network%20Function%20Requirements.pdf) Section 6.2.12 and Section 6.3.3
 
-### diagnostic
-
-#### cluster-csi-info
-
-Property|Description
----|---
-Test Case Name|cluster-csi-info
-Test Case Label|diagnostic-cluster-csi-info
-Unique ID|http://test-network-function.com/testcases/diagnostic/cluster-csi-info
-Version|v1.0.0
-Description|http://test-network-function.com/testcases/diagnostic/cluster-csi-info extracts CSI driver information in the cluster.
-Result Type|informative
-Suggested Remediation|
-Best Practice Reference|[CNF Best Practice V1.2](https://connect.redhat.com/sites/default/files/2021-03/Cloud%20Native%20Network%20Function%20Requirements.pdf) Section 6.3.6
-#### clusterversion
-
-Property|Description
----|---
-Test Case Name|clusterversion
-Test Case Label|diagnostic-clusterversion
-Unique ID|http://test-network-function.com/testcases/diagnostic/clusterversion
-Version|v1.0.0
-Description|http://test-network-function.com/testcases/diagnostic/clusterversion Extracts OCP versions from the cluster.
-Result Type|informative
-Suggested Remediation|
-Best Practice Reference|[CNF Best Practice V1.2](https://connect.redhat.com/sites/default/files/2021-03/Cloud%20Native%20Network%20Function%20Requirements.pdf) Section 6.3.6
-#### extract-node-information
-
-Property|Description
----|---
-Test Case Name|extract-node-information
-Test Case Label|diagnostic-extract-node-information
-Unique ID|http://test-network-function.com/testcases/diagnostic/extract-node-information
-Version|v1.0.0
-Description|http://test-network-function.com/testcases/diagnostic/extract-node-information extracts informational information about the cluster.
-Result Type|informative
-Suggested Remediation|
-Best Practice Reference|[CNF Best Practice V1.2](https://connect.redhat.com/sites/default/files/2021-03/Cloud%20Native%20Network%20Function%20Requirements.pdf) Section 6.3.6
-#### list-cni-plugins
-
-Property|Description
----|---
-Test Case Name|list-cni-plugins
-Test Case Label|diagnostic-list-cni-plugins
-Unique ID|http://test-network-function.com/testcases/diagnostic/list-cni-plugins
-Version|v1.0.0
-Description|http://test-network-function.com/testcases/diagnostic/list-cni-plugins lists CNI plugins
-Result Type|normative
-Suggested Remediation|
-Best Practice Reference|[CNF Best Practice V1.2](https://connect.redhat.com/sites/default/files/2021-03/Cloud%20Native%20Network%20Function%20Requirements.pdf) Section 6.2.4 and 6.3.7
-#### nodes-hw-info
-
-Property|Description
----|---
-Test Case Name|nodes-hw-info
-Test Case Label|diagnostic-nodes-hw-info
-Unique ID|http://test-network-function.com/testcases/diagnostic/nodes-hw-info
-Version|v1.0.0
-Description|http://test-network-function.com/testcases/diagnostic/nodes-hw-info list nodes HW info
-Result Type|normative
-Suggested Remediation|
-Best Practice Reference|[CNF Best Practice V1.2](https://connect.redhat.com/sites/default/files/2021-03/Cloud%20Native%20Network%20Function%20Requirements.pdf) Section 6.2
-
 ### lifecycle
 
 #### container-shutdown

README.md

@@ -161,21 +161,21 @@ There are several required arguments:
 
 * `-t` gives the local directory that contains tnf config files set up for the test.
 * `-o` gives the local directory that the test results will be available in once the container exits. This directory must exist in order for the claim file to be written.
-* `-f` gives the list of suites to be run, space separated.
 
 Optional arguments are:
-
+* `-f` gives the list of suites to be run, space separated.
+* `-s` gives the name of tests that should be skipped. This flag is discarded if no `-f` was set.
 * `-i` gives a name to a custom TNF container image. Supports local images, as well as images from external registries.
 * `-k` gives a path to one or more kubeconfig files to be used by the container to authenticate with the cluster. Paths must be separated by a colon.
 * `-n` gives the network mode of the container. Defaults set to `host`, which requires selinux to be disabled. Alternatively, `bridge` mode can be used with selinux if TNF_CONTAINER_CLIENT is set to `docker` or running the test as root. See the [docker run --network parameter reference](https://docs.docker.com/engine/reference/run/#network-settings) for more information on how to configure network settings.
-* `-s` gives the name of tests that should be skipped
 
+If `-f` is not specified, the tnf will run in 'diagnostic' mode. In this mode, no test case will run: it will only get information from the cluster (PUTs, CRDs, nodes info, etc...) to save it in the claim file. This can be used to make sure the configuration was properly set and the autodiscovery found the right pods/crds...
 
 If `-k` is not specified, autodiscovery is performed.
 The autodiscovery first looks for paths in the `$KUBECONFIG` environment variable on the host system, and if the variable is not set or is empty, the default configuration stored in `$HOME/.kube/config` is checked.
 
 ```shell script
-./run-tnf-container.sh -k ~/.kube/config -t ~/tnf/config -o ~/tnf/output -f diagnostic access-control -s access-control-host-resource-PRIVILEGED_POD
+./run-tnf-container.sh -k ~/.kube/config -t ~/tnf/config -o ~/tnf/output -f networking access-control -s access-control-host-resource-PRIVILEGED_POD
 ```
 
 See [General tests](#general-tests) for a list of available keywords.
@@ -210,7 +210,7 @@ docker build -t test-network-function:v1.0.5 \
 To make `run-tnf-container.sh` use the newly built image, specify the custom TNF image using the `-i` parameter.
 
 ```shell script
-./run-tnf-container.sh -i test-network-function:v1.0.5 -t ~/tnf/config -o ~/tnf/output -f diagnostic access-control
+./run-tnf-container.sh -i test-network-function:v1.0.5 -t ~/tnf/config -o ~/tnf/output -f networking access-control
 ```
  Note: see [General tests](#general-tests) for a list of available keywords.
 
@@ -276,13 +276,15 @@ script.
 Run any combination of the suites keywords listed at in the [General tests](#general-tests) section, e.g.
 
 ```shell script
-./run-cnf-suites.sh -f diagnostic
-./run-cnf-suites.sh -f diagnostic lifecycle
-./run-cnf-suites.sh -f diagnostic networking operator
-./run-cnf-suites.sh -f diagnostic platform-alteration
-./run-cnf-suites.sh -f diagnostic lifecycle affiliated-certification operator
+./run-cnf-suites.sh -f lifecycle
+./run-cnf-suites.sh -f networking lifecycle
+./run-cnf-suites.sh -f operator networking
+./run-cnf-suites.sh -f networking platform-alteration
+./run-cnf-suites.sh -f networking lifecycle affiliated-certification operator
 ```
 
+As with "run-tnf-container.sh", if `-f` is not specified here, the tnf will run in 'diagnostic' mode. See [Run the tests](#run-the-tests) section for more info.
+
 By default the claim file will be output into the same location as the test executable. The `-o` argument for
 `run-cnf-suites.sh` can be used to provide a new location that the output files will be saved to. For more detailed
 control over the outputs, see the output of `test-network-function.test --help`.
@@ -337,7 +339,6 @@ Suite|Test Spec Description|Minimum OpenShift Version
 ---|---|---
 `access-control`|The access-control test suite is used to test  service account, namespace and cluster/pod role binding for the pods under test. It also tests the pods/containers configuration.|4.6.0
 `affiliated-certification`|The affiliated-certification test suite verifies that the containers and operators listed in the configuration file or used by the CNF are certified by Redhat|4.6.0
-`diagnostic`|The diagnostic test suite is used to gather node information from an OpenShift cluster.  The diagnostic test suite should be run whenever generating a claim.json file.|4.6.0
 `lifecycle`| The lifecycle test suite verifies the pods deployment, creation, shutdown and  survivability. |4.6.0
 `networking`|The networking test suite contains tests that check connectivity and networking config related best practices.|4.6.0
 `operator`|The operator test suite is designed to test basic Kubernetes Operator functionality.|4.6.0
@@ -541,7 +542,7 @@ output.
 For example:
 
 ```shell script
-TNF_DEFAULT_BUFFER_SIZE=32768 ./run-cnf-suites.sh -f diagnostic
+TNF_DEFAULT_BUFFER_SIZE=32768 ./run-cnf-suites.sh -f networking
 ```
 
 # Testing certified operator

pkg/config/autodiscover/autodiscover_debug.go

@@ -59,7 +59,7 @@ func FindDebugPods(tp *configsections.TestPartner) {
 
 // AddDebugLabel add debug label to node
 func AddDebugLabel(nodeName string) {
-	log.Info("add label", nodeLabelName, "=", nodeLabelValue, " to node ", nodeName)
+	log.Info("add label ", nodeLabelName, "=", nodeLabelValue, " to node ", nodeName)
 	ocCommand := fmt.Sprintf(addlabelCommand, nodeName, nodeLabelName, nodeLabelValue)
 	_ = utils.ExecuteCommandAndValidate(ocCommand, ocCommandTimeOut, interactive.GetContext(expectersVerboseModeEnabled), func() {
 		log.Error("error in adding label to node ", nodeName)
@@ -68,7 +68,7 @@ func AddDebugLabel(nodeName string) {
 
 // AddDebugLabel remove debug label from node
 func DeleteDebugLabel(nodeName string) {
-	log.Info("delete label", nodeLabelName, "=", nodeLabelValue, "to node ", nodeName)
+	log.Info("delete label ", nodeLabelName, "=", nodeLabelValue, "to node ", nodeName)
 	ocCommand := fmt.Sprintf(deletelabelCommand, nodeName, nodeLabelName)
 	_ = utils.ExecuteCommandAndValidate(ocCommand, ocCommandTimeOut, interactive.GetContext(expectersVerboseModeEnabled), func() {
 		log.Error("error in removing label from node ", nodeName)

pkg/config/config.go

@@ -356,6 +356,7 @@ func (env *TestEnvironment) createContainerMapWithOcSession(containers []configs
 	containerMap := make(map[configsections.ContainerIdentifier]*configsections.Container)
 	for i := range containers {
 		c := &containers[i]
+		log.Debugf("Creating shell session for pod %s - container %s (ns %s)", c.PodName, c.ContainerName, c.Namespace)
 		c.Oc = configsections.GetOcSession(c.PodName, c.ContainerName, c.Namespace, DefaultTimeout, interactive.Verbose(expectersVerboseModeEnabled), interactive.SendTimeout(DefaultTimeout))
 		containerMap[c.ContainerIdentifier] = c
 	}

run-cnf-suites.sh

@@ -65,10 +65,6 @@ function html_output() {
 }
 trap html_output EXIT
 
-
-# If no focus is set then display usage and quit with a non-zero exit code.
-[ -z "$FOCUS" ] && echo "no focus found" && usage_error
-
 FOCUS=${FOCUS%?}  # strip the trailing "|" from the concatenation
 SKIP=${SKIP%?} # strip the trailing "|" from the concatenation
 LABEL=${LABEL%?} # strip the trailing "|" from the concatenation
@@ -98,13 +94,19 @@ echo "Running with skip  '$SKIP'"
 echo "Running with label filter '$LABEL'"
 echo "Report will be output to '$OUTPUT_LOC'"
 echo "ginkgo arguments '${GINKGO_ARGS}'"
+FOCUS_STRING=""
 SKIP_STRING=""
 LABEL_STRING=""
-if [ -n "$SKIP" ]; then
-	SKIP_STRING=-ginkgo.skip="$SKIP"
-fi
-if [ -n "$LABEL" ]; then
-    LABEL_STRING=-ginkgo.label-filter="$LABEL"
+if [ -n "$FOCUS" ]; then
+    FOCUS_STRING=-ginkgo.focus="$FOCUS"
+    if [ -n "$SKIP" ]; then
+        SKIP_STRING=-ginkgo.skip="$SKIP"
+    fi
+    if [ -n "$LABEL" ]; then
+        LABEL_STRING=-ginkgo.label-filter="$LABEL"
+    fi
+else
+    echo "No test suite (-f) was set, so only diagnostic functions will run. Skip patterns (-s) and labels (-l) will be ignored".
 fi
 
-cd ./test-network-function && ./test-network-function.test -ginkgo.focus="$FOCUS" $SKIP_STRING $LABEL_STRING ${GINKGO_ARGS}
+cd ./test-network-function && ./test-network-function.test $FOCUS_STRING $SKIP_STRING $LABEL_STRING ${GINKGO_ARGS}

run-tnf-container.sh

@@ -59,21 +59,21 @@ usage() {
 	     (currently: $HOME/.kube/config).
 
 	Examples
-	  $0 -t ~/tnf/config -o ~/tnf/output -f diagnostic networking -s diagnostic-nodes-hw-info
+	  $0 -t ~/tnf/config -o ~/tnf/output -f networking access-control -s access-control-host-resource-PRIVILEGED_POD
 
 	  Because -k is omitted, $(basename $0) will first try to autodiscover local kubeconfig files.
-	  If it succeeds, the diagnostic and networking tests will be run using the autodiscovered configuration.
+	  If it succeeds, the networking and access-control tests will be run using the autodiscovered configuration.
 	  The test results will be saved to the '~/tnf/output' directory on the host.
 
-	  $0 -k ~/.kube/ABC:~/.kube/DEF -t ~/tnf/config -o ~/tnf/output -f diagnostic networking
+	  $0 -k ~/.kube/ABC:~/.kube/DEF -t ~/tnf/config -o ~/tnf/output -f access-control networking
 
 	  The command will bind two kubeconfig files (~/.kube/ABC and ~/.kube/DEF) to the TNF container,
-	  run the diagnostic and networking tests, and save the test results into the '~/tnf/output' directory
+	  run the access-control and networking tests, and save the test results into the '~/tnf/output' directory
 	  on the host.
 
-	  $0 -i custom-tnf-image:v1.2-dev -t ~/tnf/config -o ~/tnf/output -f diagnostic networking
+	  $0 -i custom-tnf-image:v1.2-dev -t ~/tnf/config -o ~/tnf/output -f access-control networking
 
-	  The command will run the diagnostic and networking tests as implemented in the custom-tnf-image:v1.2-dev
+	  The command will run the access-control and networking tests as implemented in the custom-tnf-image:v1.2-dev
 	  local image set by the -i parameter. The test results will be saved to the '~/tnf/output' directory.
 
 	Test suites

test-network-function/common/suite.go

@@ -25,15 +25,18 @@ import (
 
 var env *configpkg.TestEnvironment
 
-var _ = ginkgo.BeforeSuite(func() {
+func RemoveLabelsFromAllNodes() {
 	for name := range autodiscover.GetNodesList() {
 		autodiscover.DeleteDebugLabel(name)
 	}
+}
+
+var _ = ginkgo.BeforeSuite(func() {
 })
 
 var _ = ginkgo.AfterSuite(func() {
 	// clean up added label to nodes
-	log.Info("clean up added labels to nodes")
+	log.Info("Clean up added labels to nodes")
 	env = configpkg.GetTestEnvironment()
 	env.LoadAndRefresh()
 	for name, node := range env.NodesUnderTest {