Replies: 2 comments 6 replies
-
|
To ease into the new release process, we propose a simpler approach to initially follow until the end of this year, after which we can reassess whether to proceed with the new release process or whether the simpler approach is good enough.
|
Beta Was this translation helpful? Give feedback.
-
|
When will this start? Do PRs slated for 2.5.0 need to be adjusted? |
Beta Was this translation helpful? Give feedback.
-
|
We're looking to start this after 2.5.0, so nothing will change until then. Once 2.5.0 is out, the only difference will be setting |
Beta Was this translation helpful? Give feedback.
-
|
After meeting with various community members at APEX and discussing this proposal, I now propose making the following changes to improve the release process even further based on their feedback:
|
Beta Was this translation helpful? Give feedback.
-
|
To play devil's advocate, a lot of other ecosystems (like Stellar) do just get to really big numbers and that's fine. |
Beta Was this translation helpful? Give feedback.
-
|
Either way works for me. When we start prepping for the first non-amendment release sometime next month we can pick this up again. It's more a matter of communicating the decision we make well, I think. |
Beta Was this translation helpful? Give feedback.
-
|
Seems like in between 1 and 2 might be "update major version on releases including amendments"? That's an incompatible breaking change. Although that would probably end up updating the major version every quarter. |
Beta Was this translation helpful? Give feedback.
-
|
After further discussion, and acknowledging that high version numbers are fine, we will maintain the version numbering as described in the original proposal above, so |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
The release process was last discussed a few years ago, and we would now like to refine it even further to improve how we release code.
Summary
(X.0.0)will only include changes requiring an amendment.develop_majorbranch.release_majorbranch.(X.Y.0)will only include changes not requiring an amendment.develop_minorbranch.release_minorbranch.(X.Y.Z)will only contain hotfixes and may require an amendment if needed.masterbranch.releasebranches, whereas releases and hotfixes are cut from themasterbranch. After any kind of release has been made, the code is merged down to lower branches.1 Introduction
At the moment a new
rippledrelease is made approximately every quarter, typically varying between 2 and 4 months. A standard release contains one or two new features, as well as a slew of performance and other improvements. There is also usually a mad dash to get a few important or nice-to-have PRs merged in the last few days before a release is cut, since otherwise their functionality won't be available until the next quarterly release.Around a month is reserved for intensive testing after the release has been cut, to ensure there are no regressions, before the packages are distributed and the release is announced. Sometimes last-minute urgent fixes are merged into the release after it has been cut but before it is published. In practice, the testing time is about 1-2 weeks.
This document proposes to modify the release process to improve the release frequency and better safeguard code quality. In particular, this is achieved by separating non-amendment changes (fast release cadence) from amendment changes (slow release cadence), whereby the former can be trivially rolled back.
2 Proposed solution
We propose to keep changes requiring an amendment separate from all other changes. The former will be included in major
(X.0.0)releases, while the latter will be included in minor(X.Y.0)releases. Hotfixes are an exception to the rule and serve to address a newly discovered critical issue, which may or may not require an amendment; these are included in patch(X.Y.Z)releases.An example release schedule is shown in Figure 1, where every three weeks a minor release is cut, which then via manual and automated testing is published about two weeks later. However, every few months a major release is made, during which the minor releases are temporarily suspended. As major releases carry greater risk, three weeks are reserved for testing.
Figure 1. Example release schedule with circles on the dates indicating minor releases (solid green) and major releases (dashed red), with the cut dates in a lighter color and the release dates in a darker color. For instance, in this example schedule a major release would be cut on 17 June, which would then be made available on 7 July, while a minor release would be cut on 8 July, which would then be made available on 21 July. Although the major release date and the minor cut date are just one day apart, as development continues while a major release is being prepared, there should be sufficient changes to include in a new minor release. Note that the calendar above is for illustration purposes only, and does not account for public holidays except for the vacations around the end of the year.
The motivation for not making a new minor release while a major release is already being prepared but not yet published is because the testing process of the major release would then be evaluating non-final code. Namely, the new code in the minor release would not initially be included in the branch used for the major release, but it would still need to be included before the major release is actually made, possibly introducing bugs due to the untested interaction of the combined changes.
The release schedule can remain flexible, and can be easily adjusted as needed. For instance, if a feature is running behind, the major release can be delayed by a few weeks, possibly allowing for a new minor release to be made in the meantime, while if no meaningful new functionality has been merged then a minor release can be skipped.
2.1 Advantages
The advantages of this proposal are as follows.
2.1.1 Frequent improvements
By making more frequent releases, non-amendment improvements become available more quickly to the public and especially node operators.
2.1.2 Fewer changes
By making more frequent releases, fewer changes are included in a release. This not only allows for the release testing cycles to be shortened, but also enables them to better target the modified functionality. In contrast, at present tens of PRs can be included in a major release, which can modify all aspects of rippled and thus requires extensive and exhaustive testing.
2.1.3 Catch the next minor release
Non-amendment changes that are not ready when a minor release is cut can simply go into the next one a few weeks later. This avoids the rush to get changes in, which otherwise increases the risk of bugs and the creation of tech debt.
2.1.4 Skip minor releases
UNL validators, who tend to not want to update as often given their outsized impact on the XRPL, can focus on the major releases and skip as many minor releases as they wish – effectively maintaining the status quo. We may nonetheless still encourage them to install a minor release when that has important performance or security implications. In general, any node operator can choose to update to or skip a minor release as they see fit; however, everyone must update to a major release to avoid becoming amendment-blocked.
2.1.5 Rolling back minor releases
At present all changes go out together in a single release, which means that if one of the non-amendment changes introduced a bug we cannot roll back due to the presence of amendment-related code, in which case we can only fix forward. Fixing forward, however, may incur a two week voting delay if the fix requires another amendment during which the bug can be exploited. By separating these types of changes we can worry less, since minor releases can be trivially rolled back.
2.2 Disadvantages
The disadvantages of this proposal are as follows.
2.2.1 Automating the release process
Our current release process is mostly manual. To reduce the burden on the maintainers when making a new release every few weeks, the process must be automated as much as possible. This includes (i) turning the code into artifacts, (ii) testing the artifacts, and (iii) releasing the artifacts, and (iv) automating generation of the release notes. Although automating the release process will be a good thing, getting to that point will demand a considerable time investment.
2.2.2 Juggling branches and commits
Making more frequent releases requires juggling more branches. This will involve more extensive use of git, including cherry-picking commits, reverse-merging, and resolving conflicts, which necessitates creating new processes for the maintainers to follow. However, we aim to provide helper scripts to cover the vast majority of situations, and leave the rest to automations via GitHub CI.
2.2.3 Upgrade fatigue
Communication about the releases will need to be improved to avoid update fatigue, for those not taking advantage of the auto-update scripts. In that situation, node operators will need to be notified through other channels so they can perform the update. In particular, it is easy to notice an announcement every quarter, while if they happen every two to three weeks they may be easier to miss, or node operators may simply skip them as they may think the benefits do not outweigh the effort of updating.
2.2.4 Release notes and documentation
The new release process will demand more time for preparing release notes and documentation. As new features will only be released every quarter, which require updated documentation, we do not expect a significant increase in burden. However, occasional performance / architecture improvements that will be published as a minor release may warrant updates to the documentation. Release notes are already mostly auto-generated via GitHub.
2.2.5 Close collaborations with other XRPLF developers
With more frequent releases there may be a need for closer collaboration with the developers responsible for Clio,xrpl.js, etc., especially in terms of aligning changes made to their codebases and ensuring their release schedule matches ours for important changes.
2.2.6 Challenging troubleshooting
Once we start releasing more frequently, it is likely we will have a diversity of versions running on the network, both UNL validators and non-UNL validators. As a result, when an issue surfaces it may become more difficult to pinpoint the root cause.
2.3 Discussion
By investing some time in (i) defining the branching strategy, (ii) automating our release process, (iii) educating the repository maintainers, and (iv) creating and following a tentative release schedule, we can reap the rewards of releasing faster, improving code quality and testability, reducing tech debt, and giving us more room to maneuver in case a bug is detected.
Properly executing the new release process will require giving the maintainers the strength to say "no", especially when there are incentives or pressure to deviate from the new process "to include just this one small / important PR" in a release. We expect that initially, as the new release process is being tried out, there will still be changes included in a major release that ought to have been included in a minor release. However, we also expect that over time, as the maintainers get used to the new process, the releases will get cleaner and the time between cut and publish can be shortened to release even more frequently.
3 Alternatives considered
3.1 Backward-(in)compatible changes
We can change our definitions of what is included in major and minor releases by focusing on backward-compatibility instead. In this option, backward-compatible changes are excluded from major releases, while all backward-incompatible changes are excluded from minor releases. This ensures that minor releases can always be rolled back. A major release will then still typically introduce one or more amendments while a minor release will never do so, as before. However, it is possible for a major release to not introduce any amendment, for instance when it contains a major "breaking" refactor of fundamental logic but no new features.
Discussion
While this option sounds appealing, by including non-amendment backward-incompatible changes in a major release, we may increase the amount of code that is part of a major release and would need to be tested. In contrast, in the proposed solution a backward-incompatible change can be released as a separate minor release, to allow for targeted testing of just that functionality.
Furthermore, it is not clear if changes to
rippledcan even be backward-incompatible. Since nodes upgrade to a new version independently, new releases must therefore be compatible with old releases. In particular, one can just wipe all data used by rippled, roll back to the previous release, and receive the data again from all peers.3.2 More frequent minor releases
This option is a hybrid between the status quo and the proposed solution, whereby minor releases are frequently made containing non-amendment changes, while major releases would also be allowed to contain non-amendment changes. This option would use the same release schedule as the proposed solution.
Discussion
An advantage of this option is that non-amendment changes won't have to be withheld from major releases, so development can progress more naturally.
A disadvantage is that a bug introduced by a non-amendment change included in a major release cannot be rolled back and only fixed forward, as is currently the case. Moreover, like the previous option proposed above, by including non-amendment changes in a major release, we increase the amount of code that is part of the release and needs to be tested.
3.3 Cherrypicking
Rather than juggling multiple branches, we can keep things simple with a
develop,release, andmasterbranch like now. When it is time to cut a beta or release candidate for a major or minor release, we cherrypick the corresponding commits from thedevelopbranch into thereleasebranch, and when it is time to make a release we promote the latest release candidate into themasterbranch.Discussion
Although the seeming simplicity of cherrypicking is appealing, in reality it may be difficult to do due to (i) having to tease apart the code applicable to minor releases from the code applicable to major releases, (ii) conflicts can occur in when the cherrypicked changes are subsequently changed again, and (iii) no conflicts occur when they should, resulting in code differing from what one would intuitively expect; see examples here and here.
Furthermore, we may not be able to complete a minor release when it is time for a major release. The
releasebranch will then already have one or more betas or release candidates containing non-amendment changes, which will pollute the major release containing only amendment changes we now want to make instead. In sum, cherrypicking is more trouble than it's worth.3.4 Feature branches
In this option, no amendment code is merged into the
developbranch until it is time to make a major release, allowing us to keep the existing branch structure withreleaseandmaster. Non-amendment code will be continuously merged into thedevelopbranch, enabling us to make minor releases every few weeks. It is the responsibility of the developers to frequently merge thedevelopbranch into their feature branches to keep them up to date, which may require resolving conflicts from time to time.Discussion
Code requiring an amendment is typically kept in long-lived feature branches, which already requires their developers to keep them up to date relatively frequently - the main difference proposed by this option would be that each feature branch is not merged until closer to the release date.
A typical release only includes a handful of features, so the amount of work to keep the code in the branches will generally be relatively minor. Even though minor, keeping the code in branches instead of merging them does require non-trivial effort - busywork to be precise - that could be better spent on other more useful things. Moreover, when the release date is coming up, merging multiple feature branches that have been independently developed could result in new conflicts that need to be resolved, as well as unexpected interactions between these features that may be difficult to detect and debug so close to the release date.
3.5 Do nothing
The current release process works well enough, and is not worth the time and effort to improve. Instead, we should concentrate on bringing more features to the ledger, reducing tech debt, reviewing the 60+ open PRs, and addressing the 350+ open issues.
Discussion
This document has highlighted various shortcomings of the current process, such as it consuming a lot of time from maintainers, releases containing many changes that require extensive testing, delayed availability of improvements, and so on. We believe that a short-term investment in an improved release process will in the long-term free up the developers to a larger extent, which then can be used to introduce new features, reduce tech debt, etc.
4 Implementation
Betas and release candidates are currently made by copying the latest code from the
developbranch into thereleasebranch, updating the version number, and releasing it from there. Releases are made by copying that code into themasterbranch, updating the version number once again, and then releasing it from there. Afterwards changes are merged down to ensure all lower branches have the released code. Hotfixes are made in themasterbranch, and merged down afterwards.To separately release amendment and non-amendment code, we need to slightly change our branching and release strategies. We further wish to automate the process as much as possible to reduce toil.
4.1 Terminology
Contributor
A person who creates a pull request (PR) from their private fork into the
rippledrepository.Developer
A person who can create a PR from a branch in the
rippledrepository, which requiresWriteaccess. A developer has a superset of permissions relative to a contributor.Maintainer
A person who can merge a PR into a protected branch, which requires
Maintainaccess. A maintainer has a superset of permissions relative to a developer.4.2 Branching strategy
We will distinguish between development intended for minor releases from development intended for major releases by having two parallel sets of protected branches that culminate in the master branch, see Figure 2.
Figure 2. Illustration of the branching setup. Protected branches are shown in yellow (center), while regular branches are shown in purple (top), green (bottom-left), or salmon (bottom-right) depending on whether they target a patch, minor, or major release, respectively. Changes are pushed up from the bottom minor and major branches into the higher branches, where betas and release candidates are made from either release branch, while releases are made from the master branch. After a release has been made from the master branch, the changes are merged back into the lower branches that do not yet have them. Hotfixes are merged directly into the master branch before making a patch release.
Our branching strategy uses the following protected branches.
The
masterbranchThe
masterbranch will contain the latest released code, excluding betas and release candidates. When a major, minor, or patch release has been made the code is merged down into allreleaseanddevelopbranches that do not yet have those changes.The
release_minorbranchThe
release_minorbranch will contain the code for the next pending minor release. No code will be pushed to themasterbranch until we are ready to make a minor release.The
release_majorbranchThe
release_majorbranch will contain the code for the next pending major release. No code will be pushed to themasterbranch until we are ready to make a major release.The
develop_minorbranchThe
develop_minorbranch is where development is done that leads to minor releases. This branch will be the default branch, so any created PR will use it as the base branch by default. No code will be pushed to therelease_minorbranch until we are ready to make a beta or release candidate.The
develop_majorbranchThe
develop_majorbranch is where development is done that leads to major releases. As amendment code is written less frequently than non-amendment code, the authors of any amendment PRs will need to manually select thedevelop_majorbranch as the base branch, which is a trivial step. No code will be pushed to therelease_majorbranch until we are ready to make a beta or release candidate.Hotfixes
Hotfixes are a special case and are released only when needed to fix a serious bug, and result in the creation of a patch release. A hotfix will be created on a separate branch that is based off the latest code in the
masterbranch, since thereleasebranches may have moved on. Once merged into themasterbranch, it is merged down into allreleaseanddevelopbranches. A hotfix should not need a beta or release candidate, as it should be a very targeted change that can be tested separately.In Figure 3 we show a different perspective of how the code flows between the branches, and how they are tagged.
Figure 3. An illustration of the proposed merging strategy. The green checkmarks represent PR merges into the
developbranches, while the orange targets represent PR merges into thereleaseormasterbranches that result in some type of release; the latter events are accompanied by a purple flash to highlight where the code came from. When a bug is found, a patch release is made. After each release the code is merged back from themasterbranch into allreleaseanddevelopbranches, indicated by the grayish double-arrows.4.3 Development strategy
The development process described below is shown in Figure 4.
4.3.1 Local changes
To make a change to the codebase, a developer first starts locally with a fresh branch based on the target branch.
Amendment changes
New features or other modifications that require an amendment will be merged into the
develop_majorbranch.It is the responsibility of the developer to make sure their local branch is periodically brought up to date with the
develop_majorbranch.Non-amendment changes
Any modification that does not need an amendment will be merged into the
develop_minorbranch.It is the responsibility of the developer to make sure their local branch is periodically brought up to date with the
develop_minorbranch.Betas and release candidates
Before proposing a beta or release candidate, the developer must bring their local branch, which is based on the
release_majorbranch for major releases or therelease_minorbranch for minor releases, up to date with the commits to include. This is usually achieved by pulling in the latest changes from the correspondingdevelop_majorordevelop_minorbranch, resolving conflicts as needed.To allow development to continue uninterrupted and to remain flexible when a code improvement is delivered in parts (in particular, we prefer many small PRs over few large PRs) there may be commits that we wish to skip, in which case we need to cherrypick the ones to include. Given the aforementioned challenges with cherrypicking, care must be taken when doing so.
Releases
Before proposing a release, the developer must bring their local branch, which is based on the
masterbranch, up to date with the commits to include. This is achieved by pulling in the latest changes from therelease_majorbranch for major releases or therelease_minorbranch for minor releases, resolving conflicts as needed.In general, the last release candidate from the appropriate
releasebranch contains the code that will be released, in which case no local changes actually need to be made. However, hotfixes are a special case and will require local development.4.3.2 Remote changes
Commits
When pushing a new commit to a PR, our GitHub CI workflow will execute the regular GitHub CI pipeline to build and unit test the code on Linux, MacOS, and Windows; the latter two platforms are best-effort only, i.e. we will ensure the code compiles and runs, but they are not officially supported. To merge a PR one or two approvals are needed, depending on how trivial the change is.
Merges
When a PR is merged into one of the protected branches, our GitHub CI pipeline will create the necessary Linux packages and Docker images tagged with the (shortened) commit hash, and will publish them to our internal registry. The pipeline will then set up a small yet realistic network on a test bench and execute a suite of integration tests to confirm both performance and correctness; the suite will be more comprehensive on Linux than on MacOS and Windows. PRs are not permitted to be merged into the target branch if they are out of date.
Note: Currently the
BuildInfo.cppfile contains the release version, which is a painful part of the current release process as a new commit is needed to update it, which then kicks off an entire build and test cycle. To avoid this, we will follow the lead of Clio, whereby the version is determined using Git and then injected into theBuildInfo.cppfile by CMake before the binary is rebuilt. Since only the version changes, we can skip running tests.Note: Currently the release notes are added to
RELEASENOTES.mdwhen making a release, in addition to being visible in GitHub and also included in the accompanying blog post published by our communications team afterwards. To avoid having to make an arguably superfluous change in themasterbranch that then needs to be merged down into allreleaseanddevelopbranches, we will remove this file going forward to save us time and effort.QA
Any additional testing that targets the merged changes can be performed manually at this stage. This will typically only be necessary in the
releaseandmasterbranches. We further would like to ensure that no one rebuilds the binary themselves for QA testing to avoid accidental mistakes, such as testing a different version of the code. This is why the testbed will only run using the internally hosted artifacts. As the testbed exports logs and metrics to Grafana (and in the future traces as well), they can be analyzed for regressions.Devnets
In addition to deploying to the testbed, we may consider automatically deploying betas and release candidates to devnet to be exposed to more realistic traffic. However, as major releases cannot be rolled back once amendments have been activated – unless devnet is fully reset – we require the voting to be done manually.
We must use two separate devnets, because the
release_minorandrelease_majorbranches will be developed independently. Specifically, they will diverge over time and only come back together again after each release, while new amendments proposed in therelease_majorbranch would not be available in therelease_minorbranch.To illustrate, if we were to use a single devnet then we could not first deploy a minor release candidate, e.g.
4.5.0-rc1, then a major release candidate, e.g.5.0.0-rc1, and finally deploy another minor release candidate, e.g.4.5.0-rc2, without resetting devnet in between due to the new amendments introduced by the major release candidate that the minor release candidate would not have and get blocked by.A hotfix, depending on whether they require an amendment, should be deployed on the appropriate devnet.
Figure 4. GitHub CI pipelines are triggered when adding a commit or merging a PR.
4.4 Release strategy
As the release process is quite involved and mistakes are easy to make when manually executing commands, we will automate the process to the maximum extent possible. In particular, we will create two scripts that perform the heavy lifting.
Betas and release candidates
The first script triggers a pipeline as illustrated in Figure 5.
Figure 5. GitHub CI pipelines are triggered when creating a beta or a release candidate from a
releasebranch.Pipeline
release_majororrelease_minorbranch.X.0.0-(b|rc)N.X.0.0-bN.X.0.0-b1if(X-1).Y.Zis the latest release.X.0.0-bNifX.0.0-b(N-1)is the latest beta release.X.0.0-rcN.X.0.0-rc1if(X-1).Y.Zis the latest release.X.0.0-rcNifX.0.0-rc(N-1)is the latest release candidate.X.Y.0-(b|rc)N.X.Y.0-bN.X.Y.0-b1ifX.(Y-1).Zis the latest release.X.Y.0-bNifX.Y.0-b(N-1)is the latest beta release.X.Y.0-rcN.X.Y.0-rc1ifX.(Y-1).Zis the latest release.X.Y.0-rcNifX.Y.0-rc(N-1)is the latest release candidate.X.0.0-(b|rc)Nthen therelease_majorbranch is checked, otherwise therelease_minorbranch is checked.In practice we may split the script into two scripts to distinguish between minor and major releases to avoid some of the tag matching logic described above.
Releases
The second script triggers three to five pipelines in sequence and is illustrated in Figure 6.
Figure 6. GitHub CI pipelines are triggered when creating a release from the
masterbranch. Depending on the type of release some of the PR pipelines can be skipped as they would be no-ops.Pipeline 1
{{Generated}}stanza, release notes will be auto-generated and inserted at that location.(X+1).0.0,X.(Y+1).0, orX.Y.(Z+1)withX.Y.Zbeing an existing release.Pipelines 2, 3, 4, and 5
masterbranch down into allreleaseanddevelopbranches.masterbranch, some conflicts may need to be manually resolved.releaseanddevelopbranches.release_majoranddevelop_majorbranches.release_minoranddevelop_minorbranches.In practice we may split the script into three scripts to distinguish between patch, minor and major releases to avoid some of the tag matching logic and simplify the PR pipelines described above.
5 Conclusion
By separating development into amendment and non-amendment changes we will be able to make more frequent releases, which are smaller and therefore easier to comprehensively test, which ultimately should increase code quality, reliability, robustness, and security.
By further automating the process as much as possible we would greatly reduce the reliance on manual effort to create betas, release candidates, and releases, leaving developers with more time to spend on value-add contributions.
Some manual efforts would be needed after a release has been made to update lower branches, but we believe that with smaller releases the efforts to resolve conflicts and to merge down the changes will be manageable.
Beta Was this translation helpful? Give feedback.
All reactions