cmd/go, cmd/distpack: build and run tools that are not necessary for builds as needed and don't include in binary distribution

[matloob]

Proposal Details

This proposal is to stop including tools that are not needed for builds in the binary distribution. Instead the tools would be built and run as needed by go tool using a similar mechanism to that used to build and run tools declared with a tool directive in go.mod. The goal of this proposal is to reduce the go binary distribution size.

In cmd/distpack, we should be able to discard the following tools before packaging the binary distribution: addr2line, buildid, nm, objdump, pprof, test2json, and trace. These tools do not seem to be invoked by the go command. We could go a little further and also remove doc and fix too because they are only invoked by go doc and go fix respectively.

The install target for main packages in cmd will not change, so go install will continue to install binaries to $GOROOT/pkg/tool/$GOOS_$GOARCH.

If a tool binary is present in that location, it will be run from there, but if not, the go command will fall back to building (if the tool binary isn't cached) and running the tool, similarly to how go tool builds and runs tools declared with the tool directive.

Potential issues we should watch out for and which I would be interested in hearing feedback about:

• As @prattmic pointed out, users who install a tool, make changes to the source, and then try to use go tool to run it may be surprised that the tool that's run doesn't include their changes. So we may want to think a little more about what to do in that case.
• If there any use cases that depend on running the (non-build) tools directly from the pkg/tool/$GOOS_$GOARCH directory? This could potentially break them.
• If there are use cases that depend on running the (non-build) tools on systems that have limitations on the set of binaries allowed run on the system and won't allow users to build and run their own binaries. These are systems that go test and go run wouldn't work on either.

cc @dmitshur

[mvdan]

I agree with the overall goal of this issue, but I wanted to note that currently go build -x does mention at least one of these tools directly by path, e.g. from one of my main package builds:

/home/mvdan/tip/pkg/tool/linux_amd64/buildid -w $WORK/b001/_pkg_.a # internal

Perhaps it is enough to rewrite the -x mechanism to show go tool buildid instead. It is likely worth searching through the rest of the codebase to see if these tools are mentioned elsewhere.

[seankhliao]

Does it need to change to go tool buildid? If it's the resolved path then it seems using the path to the cached binary would be more accurate for debugging / reproducing issues, especially if you have a bug that affects the tools.

[mvdan]

@seankhliao if the path gets changed to point to a cached binary I guess that would also be fine, but that would be a pretty long and random path, and it wouldn't work if the cache isn't warm - for example if it gets cleaned up manually or after a few days. I would personally prefer the shorter form; it will reproduce the build faithfully as long as none of the inputs like the source code is changed. I don't have a strong opinion here, but -x needs to change one way or another.

@matloob personally, out of all the tools you list, I've only ever run pprof, trace, and objdump directly. I don't think I've ever ran any of them directly via a full path.

users who install a tool, make changes to the source, and then try to use go tool to run it may be surprised that the tool that's run doesn't include their changes. So we may want to think a little more about what to do in that case.

Could you clarify whether the source for these tools would still be shipped? That is, that the first invocation of e.g. go tool nm would build the local source and run a cached binary, but it wouldn't download anything.

If that's the case, wouldn't changing a tool's source code cause go tool to rebuild it thanks to the build cache's hashing of build inputs?

[ianlancetaylor]

@mvdan The #internal at the end of that line is meant to be a hint that that cmd/go is doing the effect of build, but it is not actually running buildid. The buildid command is a thin wrapper around the "cmd/internal/buildid" package, and the cmd/go tool actually just calls "cmd/internal/buildid"; it doesn't run the buildid program itself.

[matloob]

@mvdan Yes, to preserve the ability to run scripts produced by go build -x, we should list go tool buildid instead of the path to buildid. There's a script test that tests this (cmd/go/testdata/script/build_dash_x.txt).

Could you clarify whether the source for these tools would still be shipped? That is, that the first invocation of e.g. go tool nm would build the local source and run a cached binary, but it wouldn't download anything.

Yes, the source for the tools would still be shipped in GOROOT. The invocation of the tool wouldn't download anything.

If that's the case, wouldn't changing a tool's source code cause go tool to rebuild it thanks to the build cache's hashing of build inputs?

If we default to running a tool that's present in $GOROOT/pkg/tool/$GOOS_$GOARCH instead of rebuilding it every time, then we might run an old version of it that doesn't take the latest changes into account. A user would have to reinstall the tool (or clean it) to get the behavior of running the latest tool. The reason I think we should still consider doing this is that some folks may want to have pre-build tools always available so that they don't have to be rebuilt if they expire in the cache.

[aclements]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— aclements for the proposal review group

[aclements]

As @prattmic pointed out, users who install a tool, make changes to the source, and then try to use go tool to run it may be surprised that the tool that's run doesn't include their changes. So we may want to think a little more about what to do in that case.

Maybe I'm misunderstanding this, but isn't this true today, too? If I change the source to cmd/compile and run go tool compile, it doesn't rebuild the compiler. (This behavior is important for things like toolstash, but I think only matters for the build tools, and we could probably accomplish that some other way if necessary.)

[aclements]

How much would this save in the download size (in particular, after compression)? At a glance, it looks like a fair amount, but it would be good to have some hard numbers.