Tracking common test flakes

[phlogistonjohn]

Without formally committing tracking particular flakes to an issue, I'd like to gather the common causes of test failures in one place, so a discussion seems to make sense. Please add repeat test failures you see to this discusssion, thanks!

[phlogistonjohn]

NT_STATUS_IO_TIMEOUT occurs running smbclient.

=== RUN   TestIntegration/smbShares/smbSharesInDefault/TestShareAccessByServiceName/TestLogin
    share_access_test.go:41: 
        	Error Trace:	share_access_test.go:41
        	Error:      	Received unexpected error:
        	            	failed to execute smbclient command: ['smbclient' '-Usambauser%1nsecurely' '//tshare3.default.svc.cluster.local/My Other Share' '-c' 'ls']: failed executing command (pod:samba-operator-system/smbclient container:): command terminated with exit code 1 [exit: 1; stdout: ; stderr: do_connect: Connection to tshare3.default.svc.cluster.local failed (Error NT_STATUS_IO_TIMEOUT)
        	            	]
        	Test:       	TestIntegration/smbShares/smbSharesInDefault/TestShareAccessByServiceName/TestLogin

[spuiuk]

PR #215 had two errors. One of them was

=== RUN TestIntegration/smbShares/users1/TestShareAccessByIP/TestLogin
share_access_test.go:42:
Error Trace: share_access_test.go:42
Error: Received unexpected error:
failed to execute smbclient command: ['smbclient' '-Usambauser%1nsecurely' '//10.42.0.20/My Share' '-c' 'ls']: failed executing command (pod:samba-operator-system/smbclient container:): command terminated with exit code 1 [exit: 1; stdout: ; stderr: do_connect: Connection to 10.42.0.20 failed (Error NT_STATUS_HOST_UNREACHABLE)
]
Test: TestIntegration/smbShares/users1/TestShareAccessByIP/TestLogin
=== RUN TestIntegration/smbShares/users1/TestShareAccessByIP/TestPutFile
share_access_test.go:57:
Error Trace: share_access_test.go:57
Error: Received unexpected error:
failed to execute smbclient command: ['smbclient' '-Usambauser%1nsecurely' '//10.42.0.20/My Share' '-c' 'put profile.jpeg']: failed executing command (pod:samba-operator-system/smbclient container:): command terminated with exit code 1 [exit: 1; stdout: ; stderr: do_connect: Connection to 10.42.0.20 failed (Error NT_STATUS_HOST_UNREACHABLE)
]

followed by the "too many pods match selector" error.

[phlogistonjohn]

Timeouts. Seen at least three times this week.

    smb_share_test.go:58: 
        	Error Trace:	smb_share_test.go:58
        	            				suite.go:148
        	Error:      	Received unexpected error:
        	            	too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [pod(samba-operator-system/tshare1-6ff8748f49-28n8m, ready=false)])
        	Test:       	TestIntegration/smbShares/users1/TestMetricsOnPod
        	Messages:   	smb server pod is not ready
=== RUN   TestIntegration/smbShares/users1/TestPodsReady
    smb_share_test.go:57: 
        	Error Trace:	smb_share_test.go:57
        	            				suite.go:148
        	Error:      	Received unexpected error:
        	            	too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
        	Test:       	TestIntegration/smbShares/users1/TestPodsReady
        	Messages:   	smb server pod does not exist
=== RUN   TestIntegration/smbShares/users1/TestShareAccessByIP
    smb_share_test.go:57: 
        	Error Trace:	smb_share_test.go:57
        	            				suite.go:148
        	Error:      	Received unexpected error:
        	            	too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
        	Test:       	TestIntegration/smbShares/users1/TestShareAccessByIP
        	Messages:   	smb server pod does not exist
=== RUN   TestIntegration/smbShares/users1/TestShareAccessByServiceName
    smb_share_test.go:57: 
        	Error Trace:	smb_share_test.go:57
        	            				suite.go:148
        	Error:      	Received unexpected error:
        	            	too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
        	Test:       	TestIntegration/smbShares/users1/TestShareAccessByServiceName
        	Messages:   	smb server pod does not exist

[spuiuk]

Similar errors when running for PR#212

=== RUN TestIntegration/smbShares/users1/TestShareAccessByServiceName
smb_share_test.go:57:
Error Trace: smb_share_test.go:57
suite.go:148
Error: Received unexpected error:
too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
Test: TestIntegration/smbShares/users1/TestShareAccessByServiceName
Messages: smb server pod does not exist
=== RUN TestIntegration/smbShares/users1/TestShareEvents
smb_share_test.go:57:
Error Trace: smb_share_test.go:57
suite.go:148
Error: Received unexpected error:
too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
Test: TestIntegration/smbShares/users1/TestShareEvents
Messages: smb server pod does not exist
=== RUN TestIntegration/smbShares/users1/TestSmbShareServerGroup
smb_share_test.go:57:
Error Trace: smb_share_test.go:57
suite.go:148
Error: Received unexpected error:
too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
Test: TestIntegration/smbShares/users1/TestSmbShareServerGroup
Messages: smb server pod does not exist

[spuiuk]

PR #214
=== RUN TestIntegration/smbShares/users1/TestShareAccessByServiceName
smb_share_test.go:57:
Error Trace: smb_share_test.go:57
suite.go:148
Error: Received unexpected error:
too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
Test: TestIntegration/smbShares/users1/TestShareAccessByServiceName
Messages: smb server pod does not exist
=== RUN TestIntegration/smbShares/users1/TestShareEvents
smb_share_test.go:57:
Error Trace: smb_share_test.go:57
suite.go:148
Error: Received unexpected error:
too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
Test: TestIntegration/smbShares/users1/TestShareEvents
Messages: smb server pod does not exist
=== RUN TestIntegration/smbShares/users1/TestSmbShareServerGroup
smb_share_test.go:57:
Error Trace: smb_share_test.go:57
suite.go:148
Error: Received unexpected error:
too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
Test: TestIntegration/smbShares/users1/TestSmbShareServerGroup
Messages: smb server pod does not exist

[spuiuk]

PR #215 had two different failures. One of them was

=== RUN TestIntegration/smbShares/users1/TestShareAccessByServiceName
smb_share_test.go:57:
Error Trace: smb_share_test.go:57
suite.go:148
Error: Received unexpected error:
too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
Test: TestIntegration/smbShares/users1/TestShareAccessByServiceName
Messages: smb server pod does not exist
=== RUN TestIntegration/smbShares/users1/TestShareEvents
smb_share_test.go:57:
Error Trace: smb_share_test.go:57
suite.go:148
Error: Received unexpected error:
too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
Test: TestIntegration/smbShares/users1/TestShareEvents
Messages: smb server pod does not exist
=== RUN TestIntegration/smbShares/users1/TestSmbShareServerGroup
smb_share_test.go:57:
Error Trace: smb_share_test.go:57
suite.go:148
Error: Received unexpected error:
too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
Test: TestIntegration/smbShares/users1/TestSmbShareServerGroup
Messages: smb server pod does not exist

[anoopcs9]

Most common failures from CentOS CI:

=== RUN   TestIntegration/createDelete/simple/TestCreateAndDelete
error: waiting for older pods to be cleaned: context deadline exceeded
    assertion_compare.go:313: 
        	Error Trace:	cleanup_test.go:157
        	Error:      	"3" is not greater than "3"
        	Test:       	TestIntegration/createDelete/simple/TestCreateAndDelete
        	Messages:   	[]

Another one:

=== RUN   TestIntegration/reconciliation/scaleoutCluster/TestScaleoutClusterSuite
    reconcile_test.go:151: 
        	Error Trace:	reconcile_test.go:151
        	Error:      	Not equal: 
        	            	expected: 3
        	            	actual  : 2
        	Test:       	TestIntegration/reconciliation/scaleoutCluster/TestScaleoutClusterSuite
        	Messages:   	Clustersize not as expected

[anoopcs9]

Most common failures from CentOS CI:

=== RUN   TestIntegration/createDelete/simple/TestCreateAndDelete
error: waiting for older pods to be cleaned: context deadline exceeded
    assertion_compare.go:313: 
        	Error Trace:	cleanup_test.go:157
        	Error:      	"3" is not greater than "3"
        	Test:       	TestIntegration/createDelete/simple/TestCreateAndDelete
        	Messages:   	[]

With all recent cleanup and improvements to tests, above failure is no longer seen.

Another one:

=== RUN   TestIntegration/reconciliation/scaleoutCluster/TestScaleoutClusterSuite
    reconcile_test.go:151: 
        	Error Trace:	reconcile_test.go:151
        	Error:      	Not equal: 
        	            	expected: 3
        	            	actual  : 2
        	Test:       	TestIntegration/reconciliation/scaleoutCluster/TestScaleoutClusterSuite
        	Messages:   	Clustersize not as expected

But happened to notice this again.

[phlogistonjohn]

OK. Thanks for the heads up. I'm very fuzzy on what that one could mean, so I won't be jumping on that one with a patch right away.

[synarete]

#213 make explicit ImagePullPolicy=IfNotPresent for samba-pods (by default; may be configured). On my minikube setup it fixed those smb server pod does not exist failures, but I admit that it is a bit of shot-in-the-dark.

[synarete]

Unfortunately, it did not fix the failures in CI :(
Maybe it would be better to keep #213 for v0.3 or as draft.