Replace sql driver to use github.com/mattn/go-sqlite3 that requires CGO #1385
Description
Currently, SQL storage indexer (in technical preview) uses as the library/driver for the SQLite operations modernc.org/sqlite . This was chosen since it didn't require to enable CGO.
Running some local benchmarks with the SQL storage indexer, replacing this library by github.com/mattn/go-sqlite3 shows improvements specially in secs/op:
- Benchmark updating index benchmark
goos: linux
goarch: amd64
pkg: github.com/elastic/package-registry/internal/storage
cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
│ ../package-registry-main/sql_prod_indexer_main_update.txt │ sql_prod_indexer_mattn_update.txt │
│ sec/op │ sec/op vs base │
SQLIndexerUpdateIndex-16 16.65 ± 1% 14.36 ± 1% -13.74% (p=0.000 n=10)
│ ../package-registry-main/sql_prod_indexer_main_update.txt │ sql_prod_indexer_mattn_update.txt │
│ B/op │ B/op vs base │
SQLIndexerUpdateIndex-16 5.039Gi ± 0% 5.046Gi ± 0% +0.14% (p=0.000 n=10)
│ ../package-registry-main/sql_prod_indexer_main_update.txt │ sql_prod_indexer_mattn_update.txt │
│ allocs/op │ allocs/op vs base │
SQLIndexerUpdateIndex-16 13.49M ± 0% 13.59M ± 0% +0.72% (p=0.000 n=10)
- Benchmark running get queries benchmark
goos: linux
goarch: amd64
pkg: github.com/elastic/package-registry/internal/storage
cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
│ ../package-registry-main/sql_prod_indexer_main_get.txt │ sql_prod_indexer_mattn_get.txt │
│ sec/op │ sec/op vs base │
SQLIndexerGet-16 2.105 ± 6% 2.098 ± 1% ~ (p=0.436 n=10)
│ ../package-registry-main/sql_prod_indexer_main_get.txt │ sql_prod_indexer_mattn_get.txt │
│ B/op │ B/op vs base │
SQLIndexerGet-16 418.9Mi ± 0% 412.5Mi ± 0% -1.53% (p=0.000 n=10)
│ ../package-registry-main/sql_prod_indexer_main_get.txt │ sql_prod_indexer_mattn_get.txt │
│ allocs/op │ allocs/op vs base │
SQLIndexerGet-16 5.945M ± 0% 5.231M ± 0% -12.01% (p=0.000 n=10)
Benchmarking the updating of the index process, there is a significant improvement in secs/op (an improvement of -13.74%). And there is a slight increase in memory usage B/op +0.14% and allocs/op +0.72%.
Via the second benchmark related to running Get queries, there is no significant improvement regarding the time (`sec/op), but there is some improvement in memory usage both B/op and allocs/op (-1.53% and -12.01% respectively).
A part from the corresponding changes in the SQL storage indexer code, it also requires:
- updating the docker build step.
- This step must build a multi-platform docker
- at the moment of writing this issue, the platforms are
linux/amd64andlinux/arm64.
- at the moment of writing this issue, the platforms are
- These docker images should be based on chainguard/wolfi images.
- Related files:
- This step must build a multi-platform docker
- updating the Buildkite step to install
gcc(viamingw) on Windows agent:
Related PR with some initial changes: #1339
Related docs about compiling github.com/mattn/go-sqlite3 for ARM: https://github.com/mattn/go-sqlite3?tab=readme-ov-file#arm