Editor and tool integration

NOTE: This is a design document for building editor and tool integration. For instructions on setting up editors for Bazel+Go projects, see Editor setup.

Overview

Bazel users should be able to edit Go source with the full support of editors just as users of the Go command (go build, etc.) are able to.

Many editor features require deep integration with the underlying build system. For example, "Go To Definition" involves discovering which package a file belongs to, then loading type information for that package and its dependencies. Traditionally, this work has been done by small tools like godef which have hard-coded assumptions about the underlying build system that do not apply to Bazel.

The Go Tools Team has built a new framework around the Language Server Protocol which should enable better integration for all editors, regardless of the build system in use. See Rebecca Stambler's GopherCon 2019 talk go pls stop breaking my editor for an overview of this framework.

The new framework consists of the following layers:

• Editor: starts and runs the LSP server as a separate process. Sends messages to LSP server as the user edits a file or runs commands.
• gopls: official LSP server for Go. Implements commands such as "Go To Definition". Tells the editor to display errors and lint warnings. Keeps metadata such as file locations and type information in memory.
• golang.org/x/tools/go/packages: Framework for loading metadata about Go packages. Clients ask for information about packages named by command line arguments understood by the underlying build system.
• gopackagesdriver: An optional tool invoked by golang.org/x/tools/go/packages. If present, calls to packages.Load will be delegated to this tool. Requests and responses are encoded as JSON, passed over stdin and stdout. If not present, go/packages uses an internal driver that works with go list.
• Build system: The system used to build packages from the source being edited. For example, Bazel, Buck, or go build.

This document is a design for a new gopackagesdriver binary that integrates Bazel+rules_go with golang.org/x/tools/go/packages. Once implemented, tools built on go/packages (like gopls) will work with Bazel.

Driver Interface

The driver must be built and installed in PATH or named explicitly with the GOPACKAGESDRIVER environment variable.

The configuration is passed to the driver as a JSON packages.driverRequest object written on stdin.

// driverRequest is used to provide the portion of Load's Config that is needed by a driver.
type driverRequest struct {
	Mode LoadMode `json:"mode"`
	// Env specifies the environment the underlying build system should be run in.
	Env []string `json:"env"`
	// BuildFlags are flags that should be passed to the underlying build system.
	BuildFlags []string `json:"build_flags"`
	// Tests specifies whether the patterns should also return test packages.
	Tests bool `json:"tests"`
	// Overlay maps file paths (relative to the driver's working directory) to the byte contents
	// of overlay files.
	Overlay map[string][]byte `json:"overlay"`
}

The list of targets is passed to the driver as a list of command line arguments in the syntax of the underlying build system. For example, a target might be @com_github_pkg_errors//:go_default_library for Bazel.

The command line arguments may also include queries that start with file=. For such a query, the driver should return the package that includes the given source file.

The driver should print results on stdout as a JSON packages.driverResponse object.

// driverResponse contains the results for a driver query.
type driverResponse struct {
	// Sizes, if not nil, is the types.Sizes to use when type checking.
	Sizes *types.StdSizes

	// Roots is the set of package IDs that make up the root packages.
	// We have to encode this separately because when we encode a single package
	// we cannot know if it is one of the roots as that requires knowledge of the
	// graph it is part of.
	Roots []string `json:",omitempty"`

	// Packages is the full set of packages in the graph.
	// The packages are not connected into a graph.
	// The Imports if populated will be stubs that only have their ID set.
	// Imports will be connected and then type and syntax information added in a
	// later pass (see refine).
	Packages []*Package
}

The response must include all packages that were matched by command-line arguments. If package dependencies were requested, the response must include transitive dependencies as well (but only directly matched packages should appear in the Roots list). Note that the ID field of each Package is specific to the underlying build system. For Bazel, it will be a Bazel label.

The driver should not exit with a non-zero status unless there was an error that prevented it from collecting data on any packages (for example, bazel was not installed). Instead, errors should be returned through the Package.Errors field for each package.

Note that the driver should produce results, even when invoked outside of a Bazel workspace. This may be implemented by recursing into packages.Load with GOPACKAGESDRIVER set to off.

Driver Implementation

The driver will be a go_binary at @io_bazel_rules_go//go/tools/gopackagesdriver.

It will run bazel with the command line arguments it was invoked with (though special handling will be needed for standard library packages, tests, and file= queries, see below). Bazel will build one .json file for each package. Each .json file will contain a JSON-serialized partial Package object. The serialized Package objects will only use relative (reproducible) paths to source files, generated files, and compiled output files; the driver is responsible for reading source files and filling in details. For rules that build multiple packages, such as go_test, multiple .json files should be produced.

Go rules don't normally produce .json package files, so these will be built using aspects (the driver will pass an --aspects option to bazel build). Several aspects will be provided to produce different sets of outputs, depending on the requested packages.LoadMode. Only one aspect will be used for any invocation.

• gopackagesdriver_files - information for NeedName and NeedFiles. The source files will be included in the output in addition to the generated .json.
• gopackagesdriver_export - additional information for NeedCompiledGoFiles and NeedExportsFile. The Go package will be compiled and included in the output. For cgo packages, this will require running cgo and C compiler, too.
• gopackagesdriver_files_deps, gopackagesdriver_export_deps - same as the above, but files from transitive dependencies will be included as well.

When the driver runs Bazel, it will use the --build_event_binary_file option to write build status information to a temporary binary proto file. The format of this file is described by build_event_stream.proto. This tells the driver whether the build was successful and where to find the output files for each package.

Once the build is complete, the driver will read the generated .json files. Relative paths to source files in these JSON files will be converted to absolute paths. The driver will load any additional information requested by the client (for example, imports or type information). It's important this information is loaded by the driver (not the builder) because only the driver has access to an overlay that may change the content of source files. The driver may also exclude fields not requested by the client (though note that go/packages will also do this).

After processing packages, the driver will print a serialized driverResponse object on stdout that includes all requested packages and, if requested, their dependencies.

Standard library

The driver needs to be able to load packages from the standard library, which will be prepared differently from non-standard packages. Standard library packages may be named on the command like using their import paths (e.g., fmt, encoding/json). All packages in the standard library may be named with the special pattern std, which is equivalent to the target @io_bazel_rules_go//:stdlib.

When the driver is asked to build standard packages, it will build @io_bazel_rules_go//:stdlib with one of the aspects listed above. This will produce a .zip file containing .json package files for all packages in the standard library. The driver will extract packages it needs from this .zip file. The same .zip file may be used for all modes. Source files will be included in the output and will not be stored in the .zip file.

file= query

When the driver receives a file= query command line argument, the driver will attempt to convert the argument into a Bazel package name, then it will build all targets in that package with one of the above aspects. Packages that don't include the queried file will be excluded from the output.

Overlays

Editors may query information about packages that have unsaved changes in source files. The unsaved changes are represented as an "overlay", which is a map from absolute paths to file contents (map[string][]byte).

When the driver parses files to load imports or type information, it should use overlay contents if present instead of source file contents. Overlays may change package names and build tags, so in general, the driver will need to filter out source files that don't belong in a package (the .json package files will include sources that may be excluded).

Overlays may include files that don't exist on disk yet, possibly in directories that have Go targets or build files. We probably can't support this. In Bazel, a Go package is defined by a go_library target (or something compatible). If there's no target, there's no package. The target can't be built if all source files are not present.

Tests

Each go_test rule represents up to three packages in addition to the library under test: an internal test package, an external test package, and a generated main package. The driver may only return information about these packages if the tests flag is set in the driverRequest.

These packages will have the same Bazel label and import path. To distinguish them in the output, we'll add a suffix to each package's ID field.

Package	Suffix
library under test	none
internal test	" [internal test]"
external test	" [external test]"
test main	" [test main]"

Note that go list has a different suffix scheme. Tools should treat the ID field as opaque, but since there's not a good way to distinguish test packages, we might need to do something similar. For a package path p, go list uses the following suffixes:

Package	Suffix
library under test	none
internal test	" [p.test]"
external test	"_test [p.test]"
test main	".test"

Aspect implementation

.json package files will be produced by one of the aspects defined alongside the driver. For each target, the aspect may format the JSON and create the file with ctx.actions.write. It should not be necessary for an aspect to create any new Bazel actions.

• gopackagesdriver_files - can produce JSON from GoSource provider. Not recursive.
• gopackagesdriver_export - can produce JSON from GoArchive provider. Not recursive.
• gopackagesdriver_*_deps - same as above, but recursive on deps and embed edges.

Standard library

The aspects above should have a special case for the standard library. We'll produce a .zip file of .json package files for each package in the standard library. A Bazel action will be used to produce this file, and it will be implemented as a new subcommand in @go_sdk//:builder. The subcommand will run go list -json std and will transform the output as needed to produce the .zip file. The same .zip file may be used in all modes. Test packages won't be included.

Tests

The aspects above should should support all packages produced by go_test. Currently, only the GoArchive provider for the generated test main package is exposed in the API. go_test should be modified to point to GoArchive or GoArchiveData for the internal and external test archives more prominently.

A .json file should be generated for each package. During analysis, we don't know which source files belong to which package, so we should include all test source files in both the internal and external packages. The driver will need to sort this out. The driver must be responsible for this because the overlay may contain changes to package names and build constraints.

Test plan

The driver may be tested using a go_bazel_test rule, which creates a small test workspace. The test will depend on a wrapper binary that passes its arguments to bazel run @io_bazel_rules_go//go/tools/gopackagesdriver within the workspace. The test will set GOPACKAGESDRIVER to the wrapper binary, then it will invoke golang.org/x/tools/go/packages.Load in various modes and compare the results against expected results.

We need to test:

• Regular targets and standard library targets.
• Various LoadMode values from go/packages. All aspects should be tested. Dependencies should be included or not. Metadata should be added or excluded as necessary.
• With and without nogo. Nogo errors should be included in the Errors field.
• cgo.
• Internal and external tests.
• Overlays should be able to modify type information for existing files.
• Overlays should be able to switch a test source file from internal to external or vice versa.
• Various build modes
  • This will likely be rewritten in the future, but we should start with cross-compilation using --platforms. Different source files should be included in the output, depending on build constraints.
• file= query should work if file exists.
• Error conditions
  • Missing targets reported as list errors.
  • Missing source files reported as list errors.
  • Syntax errors in source files reported as parse errors.
  • Type errors in source files reported as type errors.
  • Other compile errors (including nogo errors) reported as unknown errors.
• When invoked outside a Bazel workspace, golang.org/x/tools/go/packages returns results as if no driver were present.

Note on rules_go API

None of the gopackagesdriver aspects shall be considered part of the rules_go public API for now. The interface between the driver and rules_go is private and may change without notice. An installed driver may check the version of rules_go to ensure compatibility. Alternatively, we may recommend installing a script that invokes the driver via bazel run instead of installing the driver itself. This would ensure the driver is always up to date.

We may eventually make these part of the public API in order to support tools that need to use golang.org/x/tools/go/packages within an action.

Complications and open questions

• Should gopackagesdriver run Bazel in a separate output base? Bazel only allows one build per output base, so this would unblock builds in the main workspace. However, the additional local cache would consume a lot of space and memory.
• How can actions write file names to generated JSON package files? Absolute paths won't be meaningful for remote or sandboxed actions. We could restrict the actions to being local only or we could write relative paths and absolutize them in the driver.
• Can other tools based on golang.org/x/tools/go/packages consume the files produced by the gopackagesdriver output groups in Bazel actions? This would enable more code generation tools. We might need a separate gopackagesdriver for this, but hopefully we can use the same data.
• Should metadata that requires parsing source files be produced by the builder or by gopackagesdriver? If it's produced by the builder, it can be cached by Bazel, but we won't be able to include absolute paths. If it's produced by the driver, we may need to do more work locally. The driver will need to parse files in overlays at minimum, so the implementation may be simpler if the driver does most of the work.
• Should an internal test package have a different package path than the package under test?
• Should we produce .json files with aspects or output groups? Aspects let us decouple the new logic from the regular analysis and execution logic. However, they may be a little more difficult for other Bazel actions to integrate with (unless the aspects are part of the public rules_go API).
• Should we include cgo source files except in gopackagesdriver_files mode? These files can't be type checked. Probably not.