Updating dependencies

This page describes the process of updating rules declared in go_rules_dependencies, defined in repositories.bzl. If you're a user of rules_go and want to use a different version of one of the repositories declared in go_rules_dependencies, see Overriding dependencies for instructions.

When to update

Rules in go_rules_dependencies should be updated on master before each major release of rules_go. In general, we do not update dependencies on release branches (e.g., release-0.15), since these branches are intended to be stable. We only cherry-pick bug fixes onto those branches.

What versions to use

For each rule in go_rules_dependencies we generally want to pick the most recent tagged version or the tip of master if no versions are tagged. When tagged versions are available, we use an equivalent commit attribute instead of using the tag attribute. This protects us from cases where tags are changed after being published.

Make sure to leave a comment indicating what tag or branch the commit came from and the date when it was retrieved. The date should be updated even when the commit stays the same, since this indicates we checked for new commits and found none.

What rules to use

If a repository publishes source archives as release artifacts, it's best to use http_archive with a sha256 attribute to fetch those archives. This is the fastest and most secure method, since Bazel uses a shared cache for archives that have a SHA-256 sum. Unfortunately, it's rare for repositories to publish these artifacts.

The next best option is usually git_repository with a specific commit. This is slower than http_archive and less cache-friendly, but it's fairly secure and stable (assuming Git SHA-1 is hard to forge).

http_archive with a codeload.github.com URL may be used for repositories that take a long time for git_repository to clone. GitHub generates a .zip file on demand. There may be some rate limiting on these endpoints. This hasn't been a problem in the past, but I'm concerned about wide use. DO NOT use sha256 for these URLs. These files are not stable. When GitHub updates their software, minor changes in the .zip files (e.g., file ordering, alignment) can cause SHA-256 sums to change.

How to update

The tool //go/tools/releaser automates the process of updating dependencies.

To update a specific dependency:

bazel run //go/tools/releaser -- upgrade-dep -mirror=false org_golang_x_tools

To update all updateable dependencies:

bazel run //go/tools/releaser -- upgrade-dep -mirror=false all

This does the following for each upgraded dependency:

• Finds the repository in repositories.bzl by reading # releaser:upgrade-dep org repo directives.
• Looks up the highest version.
• Downloads an archive for that version.
• Uploads the archive to mirror.bazel.build unless -mirror=false was set. This requires authorization, which only Google employees have.
• Regenerates each patch in //third_party, either by applying the old patch or by running a command from a # releaser:patch-cmd directive. Patch timestamps are santized to reduce churn.
• Updates declarations in WORKSPACE and repositories.bzl.

After running releaser, you'll still need to confirm tests are passing, then commit the changes and send a PR. If -mirror=false is set, you need to create mirror request here to upload the archives to mirror.bazel.build.

Manual steps for updating a repository

To update a repository rule, we need to 1) use a new commit and 2) generate or update patches. Let's take com_github_golang_protobuf as an example, since it has multiple patches.

• Find the latest commit or tag for the repository using the guidelines above. Let's say we found aa810b61a9c79d51363740d207bb46cf8e620ed5.
• Update the repository rule with the new commit. Make sure to update the comment indicating what version this corresponds to and the current date.

    _maybe(
        git_repository,
        name = "com_github_golang_protobuf",
        remote = "https://github.com/golang/protobuf",
        commit = "aa810b61a9c79d51363740d207bb46cf8e620ed5",  # v1.2.0, as of 2018-09-28
        patches = [
            "//third_party:com_github_golang_protobuf-gazelle.patch",
            "//third_party:com_github_golang_protobuf-extras.patch",
        ],
        patch_args = ["-p1"],
        # gazelle args: -go_prefix github.com/golang/protobuf -proto disable_global
    )

• Checkout the repository in a directory named a at that version, and delete the .git directory.

git clone https://github.com/golang/protobuf a
rm -rf a/.git

• Copy a to b and run Gazelle there. Some rules require additional arguments to be passed to Gazelle. These are noted in comments.

cp -r a b
cd b
gazelle -repo_root . -go_prefix github.com/golang/protobuf -proto disable_global
cd ..

• Create a patch with the differences between a and b, setting the diff timestamp to a fixed date. Write this into third_party with the suffix -gazelle.patch.

diff -urN a b | \
  sed -E 's/^(\+\+\+ .*)20..-..-.. ..:..:............ -....$/\12000-01-01 00:00:00.000000000 -0000/g' \
>rules_go/third_party/com_github_golang_protobuf-gazelle.patch

• Sometimes, you'll need to apply old patches and make additional changes. Copy b to c, apply the old patch, make any necessary changes, then replace the old patch with a new one.

cp -r b c
cd c
patch -p1 <../rules_go/third_party/com_github_golang_protobuf-extras.patch
# ...edit edit edit...
cd ..
diff -urN b c >rules_go/third_party/com_github_golang_protobuf-extras.patch

Updating integration tests

The package //tests/integration/popular_repos contains tests from several of the golang.org/x repositories. The build file should be updated to include new tests and to remove deleted tests. There is Python script for updating the build file.

cd tests/integration/popular_repos
python popular_repos.py
bazel test :all

Frequently, new tests won't work out of the box in Bazel. This happens especially in golang.org/x/tools because fake tests are defined in testdata directories or require a working GOROOT or GOPATH, which won't be available in the test environment. Just add these to the excludes in popular_repos.py and run it again.

Updating go_googleapis

The @go_googleapis and @org_golang_google_genproto repositories both provide Google APIs. @go_googleapis contains go_proto_library targets while @org_golang_google_genproto contains go_library targets with pre-generated .pb.go files.

The googleapis repository changes quickly and has some layout idiosyncrasies, so it's hard to offer general advice on updating it. Follow the patching procedure above and update the Gazelle directives and fixes as needed.

After updating @go_googleapis, you'll probably want to update Gazelle's special cases for proto imports. Use internal/language/proto/gen/update_proto_csv.go to update internal/language/proto/proto.csv, then use internal/language/proto/gen/gen_known_imports.go to update internal/language/proto/known_imports.go, internal/language/go/known_proto_imports.go, and internal/language/go/known_go_imports.go.

@org_golang_google_genproto should be updated using a new version of Gazelle that incorporates the new dependency resolution rules above.