CLOUDP-315271: Onboard Kundukto to CI #3862

cveticm · 2025-04-29T13:32:24Z

Proposed changes

Adds new evergreen task "generate_and_upload_sbom" to generate an SBOM and package it in the release.
Adds new file build/ci/purls.txt which is a list of purls built from the binary.
This step of work does the following:

New task "generate_and_upload_sbom":
1. Creates Silkbomb tokens
2. Generates a list of purls from the linux binary
3. Generates an non-augemented SBOM in dir compliance/.
4. Creates a Kondukto token
5. Uploads the SBOM to the mongodb_mongodb-atlas-cli Kondukto project
Modification to release process
1. Includes the SBOM in release page.
New purls check
1. A new make command generates build/ci/purls.txt for visibility to devs on the implications their changes have to the sbom
2. A new EVG check that purls.txt is up-to-date

This PR does not:

Generate SSDLC report (future work of CLOUDP-315269)
Generate augmented SBOMs (future work of CLOUDP-315270)
Generate a purls list for any other OS (future work of CLOUDP-316920

Jira ticket: CLOUDP-315271

Checklist

I have signed the MongoDB CLA
I have added tests that prove my fix is effective or that my feature works
I have added any necessary documentation in document requirements section listed in CONTRIBUTING.md (if appropriate)
I have addressed the @mongodb/docs-cloud-team comments (if appropriate)
I have updated test/README.md (if an e2e test has been added)
I have run make fmt and formatted my code

build/ci/evergreen.yml

cveticm · 2025-04-29T16:47:48Z

build/package/.goreleaser.yml

@@ -55,6 +55,7 @@ archives:
      - LICENSE
      - LICENSING-NOTES.txt
      - third_party_notices/**/*
+      - compliance/**/*


This directory currently holds the generated SBOM.
It will also hold the generated SSDLC report after future work.

there are no expectations to produce this report with the release, any reason we are including it?

In the scope we have as a goal to include SSDLC report as a part of release.
The rationale is that releases provide a convenient place to store the report and enable customers to access the report on release. And then up-to-date reports with augmented SBOMs can be provided to customers on request.

I have questions about the utility of packaging it with the bin, I don't see any other binary doing it, AKO for example is commited to the repo and not shipped

Then if we really want to ship it then I wonder why not use https://goreleaser.com/customization/sbom/#usage

@fmenezes any opinion on shifting approach of where we store the report?

Just got off a call with @cveticm basically there are 3 approaches:

Store on main/master (used by https://github.com/mongodb/mongodb-atlas-kubernetes)

Store in the release package (used by ops manager)

Store in github releases page (which it would be used by CLI/LocalDev)

The reason why we wanted to do on releases page was to make it same for CLI and Local Dev, also bear in mind our policy does not specify where to store these and most projects simply upload to S3.

also bear in mind our policy does not specify where to store these

Yes and my initial point, there's no expectation to be proactive about these and only to make it available upon request

Store in the release package (used by ops manager)

Thanks for reminding me of this I downloaded OM to see where they are but since it's 2GB package I forgot to check

in ops manager they are included in the notices folder, which in the CLI is the thirdparty notice folder

I think putting the compliance report on the GitHub release page makes the most sense. It allows customers to grab the report easily before downloading any packages.
I also think that being proactive about making these reports available could reflect well on our SSDLC policy to customers.

Let me know if either of you have strong objections

Let's proceed with the releases page we might amend README/docs about it

build/ci/release.yml

build/package/generate-sbom.sh

gssbzn · 2025-04-30T12:22:39Z

build/package/generate-sbom.sh

general design comment, can we commit the purl file and make it a CI check (GH or EVG no preference) that is kept up to date, this is similar to how other repos manage this file

check the library owners check which is kind of similar

I would't go for this approach we would jump from 2 files storing dependency information (library_owners.json and go.mod) to 3 files (library_owners.json, go.mod and purls.txt).

I would prefer for the tooling to translate dependencies into purls and later json on the fly as needed.

I'm inclined to even remove library_owners.json, given only our team has code in atlasCLI since the kubernetes plugin extraction.

I'm ok to remove lib owners (same reason) but I see value on purls being committed given shipped dependencies are not the same as the ones in go.mod and this raises awareness when adding new libs and the implications, this also comes with the comment that I'd like if purls are auto generated by the precommit hook, similar to mms

I have no strong opinion here. Having the purl file committed could definitely help with making dependencies more visible, but I see @fmenezes's point about not wanting to manage yet another file.

I can create a make command to generate purls from the binary and set up a GH action to check that it's up-to-date similar to how we do the docs check.

@fmenezes, any strong feelings about this idea?

sure that is fine, we can commit it but bear in mind the extra checks

gssbzn · 2025-04-30T12:23:58Z

build/package/generate-sbom.sh

+echo "Generating PURLs..."
+cd "$WORKDIR/src/github.com/mongodb/mongodb-atlas-cli"
+
+go build -C cmd/atlas -o tmp_binary


you can use make build git is already set up to ignore the files generated by that command, and that build process is hook to generate a similar binary to what we ship with go releaser

…mand

build/ci/release.yml

fmenezes · 2025-05-01T13:13:45Z

build/ci/release.yml

+          - workdir
+        binary: make
+        args:
+          - gen-purls


I think we shouldn't do it anymore, we should make sure the file is fully updated

fmenezes · 2025-05-01T13:15:30Z

build/package/generate-sbom.sh

+# Authenticate Docker to AWS ECR
+aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 901841024863.dkr.ecr.us-east-1.amazonaws.com
+
+cd "$WORKDIR/src/github.com/mongodb/mongodb-atlas-cli"


I don't think we need this we can set working dir at evergreen function

fmenezes · 2025-05-01T13:16:36Z

build/package/generate-sbom.sh

+aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 901841024863.dkr.ecr.us-east-1.amazonaws.com
+
+cd "$WORKDIR/src/github.com/mongodb/mongodb-atlas-cli"
+mkdir ./compliance


nit: for empty folders you can commit a complicance/.gitkeep empty file to keep folder on source code

fmenezes · 2025-05-01T13:18:21Z

build/ci/release.yml

+          - AWS_SESSION_TOKEN
+          - workdir
+        binary: build/package/generate-sbom.sh 
+  "write kondukto credentials": 


I think we should merge "write kondukto credentials" and "run silkbomb" and make sure to add a rm kondukto_credentials.env in the end

cveticm · 2025-05-02T08:48:10Z

build/ci/evergreen.yml

@@ -533,6 +533,21 @@ functions:
        binary: make
        args:
          - otel
+  "check purls":


Example of check failing if purls.txt is not up-to-date:
https://spruce.mongodb.com/task/mongodb_atlas_cli_master_code_health_check_purls_patch_36f21cd8defe25ad9c89cf4d1d09f424e62cff55_681485dc659ffc00070b41ef_25_05_02_08_44_17/logs?execution=0&logtype=all

gssbzn · 2025-05-02T10:42:36Z

Makefile

+.PHONY: gen-purls
+gen-purls: # Generate purls on linux os
+	@echo "==> Generating purls"
+	GOOS=linux GOARCH=amd64 go build -trimpath -mod=readonly -o bin/atlas-linux ./cmd/atlas


why hardcode to linux?

there are some dependencies which are used only by some builds. For the moment, we will only generate purls for linux (in the interest of reducing this PRs scope) and will include generation for all builds, compiling them into one list which will be used for sbom generation in CLOUDP-316920

gssbzn · 2025-05-02T10:44:32Z

build/ci/check-purls.sh

move to /scripts

This script is only run as part of CI and other scripts live in this dir such as check-licenses.sh so IMO it makes more sense to keep this script within ./build/ci

gssbzn · 2025-05-02T10:47:52Z

Makefile

@@ -149,6 +149,15 @@ gen-docs: gen-docs-metadata ## Generate docs for atlascli commands
 	@echo "==> Generating docs"
 	go run -ldflags "$(LINKER_FLAGS)" ./tools/cmd/docs

+.PHONY: gen-purls
+gen-purls: # Generate purls on linux os


will hooking this to the pre commit be a separate task?

I didn't intended to since, as a part of CI, we do a check to see if purls.txt is up-to-date. Devs would have to manually run this command. My intent here was to give devs visual on changes to sbom because of their work.

I'm happy to include this command in pre commit. To verify @gssbzn, this would make the CI check is redundant, correct?

this would make the CI check is redundant, correct?

No cause having the pre commit installed is opt-in, and you can also do git commit --no-verify so you still need to check the files is up to date on the CI

Also just inc ase, ideally this should only run if the go.mod is modified

Makefile

github-actions · 2025-05-02T12:20:09Z

Coverage Report 📈

Branch	Commit	Coverage
CLOUDP-227276_ssdlc_phase_2	`36f21cd`	37.1%
CLOUDP-315271_onboard_silkbomb_kundukto_to_ci	`3633dd4`	37.8%
	Difference	.7%

cveticm added 2 commits April 25, 2025 17:55

Initial tasks to generate sboms for ssdlc

cb68305

Fixes sbom generation functions

dcd3a98

github-actions bot added the atlascli label Apr 29, 2025

fmenezes reviewed Apr 29, 2025

View reviewed changes

build/ci/evergreen.yml Outdated Show resolved Hide resolved

cveticm added 5 commits April 29, 2025 15:23

Tidy up

2e7a97e

Migrates ssdlc task into release process

797f1f3

Typo fixes

196ee18

Adds depends on for releaser task

61d6df8

Shellcheck fix

85e66b3

cveticm commented Apr 29, 2025

View reviewed changes

cveticm marked this pull request as ready for review April 29, 2025 16:50

cveticm requested a review from a team as a code owner April 29, 2025 16:50

gssbzn reviewed Apr 30, 2025

View reviewed changes

build/ci/release.yml Outdated Show resolved Hide resolved

gssbzn reviewed Apr 30, 2025

View reviewed changes

build/ci/release.yml Outdated Show resolved Hide resolved

gssbzn reviewed Apr 30, 2025

View reviewed changes

build/package/generate-sbom.sh Outdated Show resolved Hide resolved

Addresses pr review

0c10041

cveticm requested review from gssbzn and fmenezes April 30, 2025 10:42

cveticm changed the base branch from master to CLOUDP-227276_ssdlc_phase_2 April 30, 2025 11:45

gssbzn reviewed Apr 30, 2025

View reviewed changes

cveticm added 3 commits April 30, 2025 14:44

Addresses pr review

c735fd0

Purl moved to make command and new gitaction

05d1f50

moves compliance file to release page and uses new amke gen-purls com…

9af0c7a

…mand

github-actions bot removed the atlascli label May 1, 2025

Fix

4c93dd1

fmenezes reviewed May 1, 2025

View reviewed changes

build/ci/release.yml Show resolved Hide resolved

fmenezes reviewed May 1, 2025

View reviewed changes

cveticm added 4 commits May 1, 2025 14:42

Ensures purls are geenrates for fixed os and addresses pr review

2b92c82

Fix

6b41768

Test fix

4c9ef07

Moves purl check to evg

c654e63

cveticm commented May 2, 2025

View reviewed changes

gssbzn reviewed May 2, 2025

View reviewed changes

Makefile Outdated Show resolved Hide resolved

Adds gen-purls as pre-commit

3633dd4

fmenezes approved these changes May 2, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLOUDP-315271: Onboard Kundukto to CI #3862

CLOUDP-315271: Onboard Kundukto to CI #3862

cveticm commented Apr 29, 2025 •

edited

Loading

cveticm Apr 29, 2025

gssbzn Apr 30, 2025

cveticm Apr 30, 2025 •

edited

Loading

gssbzn Apr 30, 2025

cveticm Apr 30, 2025

fmenezes Apr 30, 2025

gssbzn Apr 30, 2025

gssbzn Apr 30, 2025

cveticm May 1, 2025

fmenezes May 1, 2025

gssbzn Apr 30, 2025

gssbzn Apr 30, 2025

fmenezes Apr 30, 2025

gssbzn Apr 30, 2025

cveticm May 1, 2025

fmenezes May 1, 2025

gssbzn Apr 30, 2025

fmenezes May 1, 2025

fmenezes May 1, 2025

fmenezes May 1, 2025 •

edited

Loading

fmenezes May 1, 2025

cveticm May 2, 2025

gssbzn May 2, 2025

cveticm May 2, 2025

gssbzn May 2, 2025

cveticm May 2, 2025

gssbzn May 2, 2025

cveticm May 2, 2025

gssbzn May 2, 2025

github-actions bot commented May 2, 2025

CLOUDP-315271: Onboard Kundukto to CI #3862

Are you sure you want to change the base?

CLOUDP-315271: Onboard Kundukto to CI #3862

Conversation

cveticm commented Apr 29, 2025 • edited Loading

Proposed changes

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cveticm Apr 30, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fmenezes May 1, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented May 2, 2025

cveticm commented Apr 29, 2025 •

edited

Loading

cveticm Apr 30, 2025 •

edited

Loading

fmenezes May 1, 2025 •

edited

Loading