Data collection fails when there are multiple failed pods of the same type

[mrajagopal]

Description

if there are multiple failed pods amongst running ones for the same pod name, the tool fails to collect:

NAME                                                 READY   STATUS      RESTARTS   AGE
nic-v5-0-nginx-ingress-controller-74749fc98b-f6cdd   0/1     Error       0          5d16h
nic-v5-0-nginx-ingress-controller-74749fc98b-hd7nh   0/1     Completed   0          54d
nic-v5-0-nginx-ingress-controller-74749fc98b-hjbx8   0/1     Completed   0          26d
nic-v5-0-nginx-ingress-controller-74749fc98b-hwdzn   1/1     Running     0          5d16h
nic-v5-0-nginx-ingress-controller-74749fc98b-rn5hp   0/1     Completed   0          5d16h
webapp-fd76d5df6-2wdm7                               1/1     Running     0          54d

Observation

Here is the observed failure:

Running job exec-nginx-ingress-version... Error: unable to upgrade connection: container not found ("nginx-ingress")
Running job exec-nginx-t... Error: unable to upgrade connection: container not found ("nginx-ingress")
Running job exec-agent-conf... Error: unable to upgrade connection: container not found ("nginx-ingress")
Running job exec-agent-version... Error: unable to upgrade connection: container not found ("nginx-ingress")
Running job crd-objects... OK

[mrajagopal]

Having multiple failed pods amongst one working pod is not unexpected.
However, what is unexpected is that the exec commands for the working pods were not captured to file due to the exec failing on at least one pod thus marking the job failed.

The logic of Job.Collect() is that it returns early when JobResult.Error is set.

This is incorrect, as there can be data from working pods.
The fix will be reorder the test for jobResults.Error after the captured results of the exec have been written to file.

[mrajagopal]

This has been addressed in commit 0da9305 on a feature branch.