[CI]: Epic: fix everything

[apostasie]

Currently known flaky tests (could be a test issue or a nerdctl issue)

Hard hitting:

• P0: panic: Stats panic (send on closed channel) #4043
• P0: advanced: CNIEnv concurrency issues #3556

Third-party:

• P0: advanced: [CI]: Windows failure on pull: UtilityVM: Incorrect function.: unknown #3524
• P0: [CI]: buildkit Internal: not found #4142

Possibly EL8 / slow environments specific:

• P1: [LEGACY] [CI]: stty output not always there (EL8 only?) #4119
• P1: [LEGACY] [CI]: ipfs /root/.ipfs/api not available #4118
• P1: [LEGACY] [CI]: TestRestartWithSignal does not seem to (always?) work correctly (EL8 only?) #4068
• P1: [LEGACY] [CI]: NoneNetworkManager is presumably flaky wrt hostname #3971

Kube specific:

• P1 (minor, but annoying): [CI]: Kube failure on commit + save #4106
  • to be confirmed this is still happening with containerd v2

Test currently disabled (link to documented issue):

• [URGENT] Regression? TestRunWithTtyAndDetached on windows appears broken on main #3437
• Is IPFS+estargz working? #3475
• see TestNetworkInspect
• test should be re-enabled: images --filter=non-existent inconsistent (docker vs. nerdctl) #3511
• test should be re-enabled: volume (probably others) filter behavior is inconsistent with Docker #3452

"Retry-them" tests that are clearly broken:

• [CI]: flaky with forceful retry: TestSyslogNetwork #4141
• [CI]: OCI runtime start failed: cannot start a container that has stopped: unknown #4143
• [LEGACY] [CI]: TestRunBindMountBind is not working well #4144
• [CI]: TestComposeUpWithExternalNetwork network failure #4146
• [CI]: TestLogsWithoutNewlineOrEOF flaky #4147

Others:

• P0: [CI]: TestRunStargz is racy #4066
• P0: something is wrong with logs: [CI]: failure nerdctl - TestCreate/logs appears to be flaky #3717
• P1: [CI]: events filter sometimes does not work #4132 - looks like a legit issue, much more likely to happen on EL8/EL9

To confirm:

• [CI]: failure docker - TestBuildWithBuildArg/EmptyArgValueOverridesDefault #3727

CI structural issues

• P1: [CI]: github cache is too aggressive on limiting requests (429) #3733
• P2: Tests that require not-rootless are getting skipped with target=docker (where the cli is not run as root) #3453

Tests to add

• Test login, push (and authenticated pull) against docker hub #3494
• Expert: Add testing for OCI hooks #3156
• -v SRC:DST:rro,rprivate should be tested in the CI #1989
• CI: test the CRI log viewer #1957
• CI: add test for compose + restart #178

Enhancements: tigron

• [CI/tigron]: investigate file descriptor leak detection as a testing feature #4029
• [CI/tigron]: investigate go routine leak detection as a testing feature #4028
• [Tigron]: roadmap and tracking epic for work on test tooling #3448

Enhancements: generic

• dockerfile specific: [CI]: git clone revamp, part II #4031
• dockerfile specific: [CI] Epic: possible enhancements to current Dockerfile #4021
• [CI]: Discussion: should we drop testing inside a container? #3858
• [CI]: ghcr.io/containerd/stargz-snapshotter:0.15.1-kind needs an arm64 variant #3845
• We should have a publish scripts for the testing images we need, and we should not depend on third party images #3480
• [CI]: Disable test retries #3303
• Add harbor to our integration testing suite? #3257
• fix unit tests so that we can race them

Performance tracking

• Latest major disruption: [CI]: State of the CI #3924
• [CI]: Tracking status #3577

Ongoing work

Evergreen playground:

• [CI]: Evergreen / Everred #4129

CI:

• [CI]: cleanup, breakout 6: separate jobs #4191
• [CI]: fix error output for target not found #4190
• [CI]: cleanup, breakout 5: separate jobs #4189
• [CI]: cleanup, breakout 7: integration tests reports #4188

Flaky-debugging work in progress:

• [DEBUGGING]: Fix EL8 / slow systems ipfs slow start #4118 #4126
• [DEBUGGING]: buildg time-out flaky issue (#4046) #4103

CI work in progress:

• [EXPERIMENTAL] Reusable workflows for CI simplification #3735
• [WINDOWS] updated busybox image & registry image #3636
• [Experimental] Windows registry #3579

[apostasie]

@AkihiroSuda could we label area CI and help wanted?

[apostasie]

@AkihiroSuda the following are ready (in order), at your convenience.

• [CI]: quality of life: minor debugging output fixes #4111
• [CI]: fix internal command flaky test (window) #4114
• [Tigron]: allow per-command environment override #4116
• Update containerd to v2.0.5 #4125
• [PRIORITY] [CI]: Fix EL8 tests execution #4127
• [Tigron]: revised Data interface #4081 <- this one is bigger