Skip to content

ReleaseCompatibility.md

Latest commit

 

History

History
103 lines (87 loc) · 4.93 KB

ReleaseCompatibility.md

File metadata and controls

103 lines (87 loc) · 4.93 KB
Raw

Releases & Compatibility

To date, we have been somewhat cavalier about making breaking changes in Fairlearn. Although we are currently pre-v1.0.0 and hence without particular commitments to compatibility (as generally understood - see e.g. the Semantic Versioning Scheme), we should work to reduce the number of breaking changes we make.

The Problem

At the time of writing, we have just released v0.4.6, which has some substantial breaking changes in the Metrics area of Fairlearn. Before it, v0.4.5 reworked a smaller subset of this functionality, caused smaller breakages. There are also further changes to Metrics planned, which may well cause further breaks (although these should be more minor). All this is obviously undesirable from a user standpoint - these look like 'patch' level releases, but are actually breaking their code. Concretely, we've had users install Fairlearn with pip, and then find themselves unable to run Notebooks from our GitHub project - not because the functionality was missing, but because it had been renamed. Moving our notebooks to being generated as part of the documentation in our examples directory will help with this, since this will result in the notebooks being versioned. However, improved backwards compatibility is still desirable.

We do not need deprecation policies as elaborate of those of SciKit-Learn or NumPy - indeed, policies such as those would be excessive for Fairlearn, given the size of our code and user bases. However, we do need to start to move in that direction, or we will not be able to grow our user base due to chaos in the code.

Support Policy

Starting with v0.5.0 we should make a commitment that anything which works at v0.n.m_0 will also work for v0.n.m so long as m >= m_0. However, we do not guarantee compatibility between n and n+1 in this scheme (although we would seek to minimise breakage). If we have to do v.n.m.post[i] releases, we will only support the final post release in the chain.

In order to support less mature functionality, we should also add a fairlearn.experimental package (see a similar namespace in SciKit-Learn). Anything in there will be subject to breaking at any time. Using an experimental namespace would not have helped with our current set of breaks, since the changes were being made to core functionality. Rather, this is to give future developers a space where they can get feedback on new functionality without immediately being committed to supporting their speculative design decisions.

At the time of writing, it appears that the dashboard code does not use namespaces. However, namespaces are supported in TypeScript, and as we develop the UX code, we should introduce a similar distinction.

To monitor the required backwards compatibility, each new release will have a build pipeline associated with it. This pipeline will run the tests associated with the release tag against the version of Fairlearn in master. In keeping with the note about post[i] releases above, the build will only run against the last release in each release branch. The build will become part of the PR Gate for master, except when we are moving from v0.n to v0.n+1.

Revised Branching and Release Policy

Our recent releases have been made from master, without making a release branch. While this approach has desirable properties, using release branches will work better with the more robust support policy described above.

For all the master and release branches, we will require a linear history in GitHub. This forbids plain merge commits, and we will prefer squash merges to rebases.

We will keep the master branch at v0.m.n.dev0 at all times, indicating that master is under active development (although we will always seek to keep master in a shippable condition). Releases will occur from branches.

To create a new release:

  1. Create a branch release/v.0.n.m
  2. Remove the dev0 suffix from the version on the release branch
    • Bump master to v0.n.m+1.dev0
  3. Run the release pipeline on the new release branch
  4. Create a GitHub Release corresponding to the new package (hopefully this can be automated)
  5. Create the new build pipeline to monitor backwards compatibility

Post Releases

We should only put out post[i] releases for essential fixes (either to algorithms or code) - no new features are allowed. In general, fixes should be made in master and individually moved to the appropriate release branch as required. The move will probably best be done with git rebase -i, in order to preserve a linear history on each branch. After the release, create an appropriate GitHub release, and update the corresponding backwards compatibility build.