x/crypto/x509roots/fallback: high, unskippable, init cost

[twpayne]

Go version

go version go1.24.3 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE='on'
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN='/home/twp/.local/bin'
GOCACHE='/home/twp/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/twp/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1612493662=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/home/twp/src/go.googlesource.com/crypto/go.mod'
GOMODCACHE='/home/twp/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/twp'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go-1.24'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/twp/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/go-1.24/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.24.3'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

I imported golang.org/x/crypto/x509roots/fallback and observed that this slowed down the startup of my CLI application by about 8ms on a fast machine:

package main

import _ "golang.org/x/crypto/x509roots/fallback"

func main() {}

$ go build -o tmp main.go

$ GODEBUG=inittrace=1 ./tmp |& awk '{print $5, $6, $2}' | sort -n | tail -n 3
0.043 ms internal/godebug
0.052 ms runtime
8.8 ms golang.org/x/crypto/x509roots/fallback

What did you see happen?

A 8.8ms delay due to the golang.org/x/crypto/x509roots/fallback package parsing a large number of certificates in its init() function. This is on a fast CPU (AMD Ryzen 8700G). This cost is paid whether or not the fallback X.509 certificates are used or not.

What did you expect to see?

Zero or near-zero startup cost for using golang.org/x/crypto/x509roots/fallback.

From looking at the code, it seems that the CPU is cost is due to parsing about 150 certificates in .PEM format in the package's un-skippable init() function.

The obvious, easy, backwards-compatible fix would be to parse the .PEM files while executing the template that generates bundle.go, instead of parsing them every time at startup.

I would be happy to submit a CL to implement this. Is it likely that such a CL would be accepted?

[dmitshur]

CC @rolandshoemaker, @FiloSottile.

[FiloSottile]

There is no good way to persist the result of parsing (which might change in future versions) into a source file.

It's a bit unfortunate SetFallbackRoots doesn't let us sync.Once the parsing, which we do for the system pool on Linux.

[twpayne]

OK, that's a shame. Feel free to close this issue if there is no good solution.

[mateusz834]

I think it is still possible to speed-up this up slightly, by dropping the PEM-encoding and storing directly in DER. I will reopen and send a CL soon.

goos: linux
goarch: amd64
pkg: golang.org/x/crypto/x509roots/fallback
cpu: AMD Ryzen 5 4600G with Radeon Graphics
            │ /tmp/before │             /tmp/after              │
            │   sec/op    │   sec/op     vs base                │
InitTime-12   1.580m ± 1%   1.007m ± 1%  -36.27% (p=0.000 n=20)

            │  /tmp/before  │              /tmp/after              │
            │     B/op      │     B/op      vs base                │
InitTime-12   1097.1Ki ± 0%   703.1Ki ± 0%  -35.91% (p=0.000 n=20)

            │ /tmp/before │             /tmp/after             │
            │  allocs/op  │  allocs/op   vs base               │
InitTime-12   10.74k ± 0%   10.03k ± 0%  -6.62% (p=0.000 n=20)

[gopherbot]

Change https://go.dev/cl/676217 mentions this issue: x509roots/fallback: store bundle certs directly in DER

[gopherbot]

Change https://go.dev/cl/676215 mentions this issue: x509roots/fallback: move parsing code to a non-generated file