Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

run upstream tests #503

Open
tianon opened this issue Feb 18, 2025 · 0 comments
Open

run upstream tests #503

tianon opened this issue Feb 18, 2025 · 0 comments

Comments

@tianon
Copy link
Member

tianon commented Feb 18, 2025

Similar to docker-library/golang#552, there are a lot of upstream tests we could run and should run somewhere, but during build doesn't necessarily make sense.

In https://salsa.debian.org/ruby-team/ruby/-/blob/5b91c2c6b64195da7110d341daba904b9d12c9d5/debian/tests/run-all, we can see that Debian does some clever things to run a lot of the upstream tests post-build via "autopkgtests" (which is very akin to how we run tests on our images after they're built -- they typically run against the built + installed packages).

The basic gist is ruby test/runner.rb, and Debian's got a pretty hefty collection of tests they exclude which will probably inspire tests we need to exclude too (especially # FIXME for now, we are excluding the tests for C extensions; couldn't figure out how to properly build them without building everything else, since we'll have the same issue with doing this post-build).

So we'll need to recreate our "download the source code" logic to get the test files back, but that's the easy part. My own very basic testing of this idea on ruby:slim comes up with "7 failures, 6 errors, 176 skips", so there's clearly some work to do here (for each of those, fixing the image, accommodating the test somehow with runtime flags/packages/etc, or building up a list of tests to skip and writing down justifications for why we think they're valid to skip/ignore).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant