Add a CI step to check significant package output changes resulting from changes to patches.nix

[sdankel]

@mitchmindtree mentioned that nix flake show can tell us which packages would be changed by refresh-manifest.sh.

We could add this to PRs to show this output to help us catch any issues with patch changes.

[mitchmindtree]

Just to clarify, in our chat earlier the nix flake show suggestion was more related to dry-running the difference in package outputs that result from a change to the patches.nix list.

The role of the refresh-manifests.sh script is a little different - namely, it simply produces the manifest files. Changes to patches.nix will never change what manifest files are generated by refresh-manifests.sh. Rather, changes to patches.nix do change the way that manifests are patched and extended before they're used to construct the package outputs.

For addressing the issue we discussed around changes to patches.nix, perhaps a better name for this issue might be something along the lines of Add a CI step to check significant package output changes resulting from changes to patches.nix.

patches.nix changes that would result in breaking builds should be caught by the existing CI, however patches.nix changes that accidentally invalidate large parts of the cache might be trickier to check for (other than spotting CI times increasing) 🤔

One approach might be to have some CI step that runs first (before the builds begin for all platforms) that diffs the output of nix show derivation for each output and checks for significant changes. E.g. if multiple old package derivations have changed, then a patch may have unintentionally changed some old packages and invalidated the cache for older versions.

A similar check could potentially be added to test changes to fliters.nix by diffing the output of nix flake show (which shows a list of all package outputs). This might help to catch cases where a filter was added that accidentally omits loads of older versions by mistake.