-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve egui_kittest screenshot change detector #5773
Comments
### Related * Fixes #8557 * Improves test introduced in #9205 * Note that this test wasn't stable against failure previously 😱, see emilk/egui#5773 for thoughts about improving these things in the future ### What Overrides from blueprint do what they're supposed to again (unclear when this broke - did it ever work correctly?)  When trying to ensure automated testing for this things started to ripple out a little bit - commit by commit review recommended.
### Related * Fixes #8557 * Improves test introduced in #9205 * Note that this test wasn't stable against failure previously 😱, see emilk/egui#5773 for thoughts about improving these things in the future ### What Overrides from blueprint do what they're supposed to again (unclear when this broke - did it ever work correctly?)  When trying to ensure automated testing for this things started to ripple out a little bit - commit by commit review recommended.
Maybe we can introduce a per-OS treshold? When testing for the other OSes the treshold could then be overriden in CI via some environment variable? |
Different hardware produce subtly different results:
https://docs.rs/egui_kittest/latest/egui_kittest/#what-do-do-when-ci--another-computer-produces-a-different-image
So we want a screenshot test that ignores small changes but detects large ones.
We should figure out a good default algorithm for this test by building up a test set of pairs of images that should either trigger or not trigger the threshold.
For instance, we recently discovered that these two images fell under our default threshold (i.e. were considered the same) for 98% of all pixels.
The screenshots above came up during the development of rerun-io/rerun#9151
That piece of work transitively broke the test in question and expected a failure but didn't get one.
It is important to note that while this was using egui_kittest's default per-pixel treshholds, it already set a "pixel-ignore-treshold" (as described in #5683) https://github.com/rerun-io/rerun/blob/6001edac74fb39d022eaeac310958a78ac791b4f/crates/viewer/re_view_spatial/tests/visible_time_range.rs#L417
Related to:
The text was updated successfully, but these errors were encountered: