Skip to content

Upgrade to TUF v2 client #3844

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 10, 2025
Merged

Upgrade to TUF v2 client #3844

merged 1 commit into from
Jun 10, 2025

Conversation

cmurphy
Copy link
Contributor

@cmurphy cmurphy commented Aug 20, 2024

Use sigstore-go's TUF client to fetch the trusted_root.json from the TUF
mirror, if available. Where possible, use sigstore-go's verifiers which
natively accept the trusted root as its trusted material. Where there is
no trusted root available in TUF or sigstore-go doesn't support a use
case, fall back to the sigstore/sigstore TUF v1 client and the existing
verifiers in cosign.

TODO:

Fixes #3548

Summary

Release Note

Documentation

@cmurphy cmurphy requested a review from jku August 20, 2024 23:53
@cmurphy
Copy link
Contributor Author

cmurphy commented Aug 20, 2024

There is still work to do here but I will be out for a couple of weeks so it might be worth getting some eyes on in the meantime.

Copy link

codecov bot commented Aug 20, 2024

Codecov Report

Attention: Patch coverage is 18.44156% with 314 lines in your changes missing coverage. Please review.

Project coverage is 34.19%. Comparing base (2ef6022) to head (de8f774).
Report is 403 commits behind head on main.

Files with missing lines Patch % Lines
pkg/cosign/verify.go 18.39% 66 Missing and 5 partials ⚠️
pkg/cosign/tuf.go 0.00% 66 Missing ⚠️
cmd/cosign/cli/verify/verify_attestation.go 0.00% 49 Missing ⚠️
cmd/cosign/cli/verify/verify.go 0.00% 29 Missing ⚠️
...cosign/cli/fulcio/fulcioverifier/fulcioverifier.go 0.00% 17 Missing and 1 partial ⚠️
pkg/cosign/tlog.go 33.33% 13 Missing and 1 partial ⚠️
cmd/cosign/cli/verify/verify_blob.go 48.00% 9 Missing and 4 partials ⚠️
cmd/cosign/cli/verify/verify_blob_attestation.go 52.00% 7 Missing and 5 partials ⚠️
cmd/cosign/cli/initialize/init.go 66.66% 6 Missing and 4 partials ⚠️
cmd/conformance/main.go 0.00% 6 Missing ⚠️
... and 5 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3844      +/-   ##
==========================================
- Coverage   40.10%   34.19%   -5.92%     
==========================================
  Files         155      211      +56     
  Lines       10044    14009    +3965     
==========================================
+ Hits         4028     4790     +762     
- Misses       5530     8612    +3082     
- Partials      486      607     +121     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cmurphy cmurphy force-pushed the upgrade-tuf branch 2 times, most recently from e59faee to 07e1c38 Compare August 21, 2024 00:12
@cmurphy cmurphy force-pushed the upgrade-tuf branch 4 times, most recently from 00b4edb to 820d6a8 Compare September 12, 2024 23:22
@jku
Copy link
Member

jku commented Sep 13, 2024

There is still work to do here but I will be out for a couple of weeks so it might be worth getting some eyes on in the meantime.

sorry, I missed this at the time: I will have a look today or monday

@jku
Copy link
Member

jku commented Sep 13, 2024

Sigstore-go provides a way to check for a trusted root and automatically use it if available, but can also fetch individual targets as needed if the provided TUF mirror does not supply a trusted_root.json.

Can you specify what cosign does with sigstore-go? I can see there are some invidual target fetches in the code but does cosign now use trusted_root.json by default?

@cmurphy
Copy link
Contributor Author

cmurphy commented Sep 13, 2024

Can you specify what cosign does with sigstore-go?

Cosign uses sigstore-go piecemeal for various things, but the goal of this PR is to adopt sigstore-go's TUF client wrapper instead of the client wrapper provided by sigstore/sigstore.

does cosign now use trusted_root.json by default?

It does not, that's part of the purpose of this PR and #3548. There is another PR in progress #3854 that adds a --trusted-root flag that I will have to adjust this PR to conform with.

sorry, I missed this at the time: I will have a look today or monday

I may now have to wait for that other PR to be fleshed out before I can make active progress on this, so don't feel rushed to review this. I can ping you again when it's in a more stable state.

Comment on lines 29 to 37
const (
// This is the root in the fulcio project.
fulcioTargetStr = `fulcio.crt.pem`
// This is the v1 migrated root.
fulcioV1TargetStr = `fulcio_v1.crt.pem`
// This is the untrusted v1 intermediate CA certificate, used or chain building.
fulcioV1IntermediateTargetStr = `fulcio_intermediate_v1.crt.pem`
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if these filenames should be configurable somewhere for private Sigstore TUF operators. The existing metadata format allows for additional targets to be added and discovered, but TUF v2 does not allow iterating over targets, so that strategy is unsupported, meaning this current diff does not necessarily support all private TUF deployments.

We could also provide a CLI utility to convert an old TUF v1 layout to a trusted_root.json and require private TUF maintainers to use it to generate a trusted root in order to support the next version of cosign. I kind of like this option as it fast tracks the adoption of the trusted root.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if these filenames should be configurable somewhere for private Sigstore TUF operators

These filenames are hardcoded in the TUF v1 code so recreating them here was an attempt to align with the old way of retrieving targets.

The existing metadata format allows for additional targets to be added and discovered, but TUF v2 does not allow iterating over targets, so that strategy is unsupported, meaning this current diff does not necessarily support all private TUF deployments.

You're right, that was an oversight on my part. I had an earlier version of this that could support discovering targets from custom metadata that I will restore.

We could also provide a CLI utility to convert an old TUF v1 layout to a trusted_root.json

We could do that as a last resort, but my hope was to maintain full backwards compatibility and ease users gently toward using trusted_root.json.

Comment on lines 82 to 133
func checkValidityPeriod(start, end time.Time) tufStatus {
now := time.Now()
if now.Before(start) {
return inactive
}
if now.After(end) {
return inactive
}
return active
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The validity period in the TrustedRoot should be compared to the timestamp provided by the timestamping service or transparency log. I understand this is not something that is known at the time that you are assembling the CheckOpts, and that makes this a hard problem, but I'm not sure we want to enforce this in this version of the code. I believe we still want to be able to verify a signature produced prior to a timestamping service's validity end date, even if the current time is after that date.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My goal here was to have something to substitute for the Active/Expired status from TUF v1 that does not seem to have an equivalent in TUF v2. Open to discarding this entirely or finding an alternative.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For what it's worth, I don't think Cosign is effectively using Active/Expired now, as Cosign is only printing a message to stdout if you are verifying an entry with an "expired" key. This isn't really helpful to the user because there's nothing wrong with using an expired key, it's if the key is used to verify something outside of its validity window, which isn't currently implemented in Cosign.

If it'd be easier to just not deal with Active/Expired, I'd be supportive of that. We can either rely on sigstore-go down the line to do that check, or we can file an issue to implement that feature in Cosign's verification API when using the trusted root file. I'd lean towards the latter in line with #3879.

@cmurphy
Copy link
Contributor Author

cmurphy commented Sep 20, 2024

@steiza , @codysoyland , @haydentherapper and I met to discuss a path forward on this and we decided to shrink the scope of this PR by removing the fetch of trusted_root.json, to be addressed separately.

This PR dropped the fetch of trusted_root.json, but it now fails conformance tests because the SCT returned during signing from p-g-i is signed by the ctfe_2022.pub key, which the sigstore-go TUF client has no way to know about. This worked fine with p-g-i when trusted_root.json was considered, but shows that this approach will break any deployment that has rotated keys relying on custom metadata. I'm starting to think that using the new client must also be synchronized with using trusted_root.json and that the whole thing should be guarded by a flag that users can opt into. Thoughts?

@steiza
Copy link
Member

steiza commented Sep 23, 2024

but shows that this approach will break any deployment that has rotated keys relying on custom metadata

Oof, what a mess we've made for ourselves. Yeah, this is what makes the cosign modernization effort we're working on so tricky.

I'm starting to think that using the new client must also be synchronized with using trusted_root.json and that the whole thing should be guarded by a flag that users can opt into.

This is what I'm thinking as well. #3854 could be updated to use CheckOpts as @codysoyland described in #3879. Then for those commands that support --trusted-root could also add something like --tuf-url + --tuf-root that would fetch the trusted_root.json for you, instead of having to supply it with --trusted-root.

@cmurphy cmurphy force-pushed the upgrade-tuf branch 2 times, most recently from f5d1221 to b6f5dc4 Compare September 24, 2024 20:32
@cmurphy
Copy link
Contributor Author

cmurphy commented Sep 24, 2024

The latest change adds back the root.NewLiveTrustedRoot TUF fetch, now guarded by environment variables so the user can opt in. Using environment variables seems to be pretty common in cosign and it makes it so all the TUF logic is concentrated in the pkg/cosign package and we don't have to plumb through command line flags. It doesn't add a TrustedMaterial field to CheckOpts because it's not needed with this approach, the TUF functions work mostly the same as they did before. This could still work with #3854 where that PR would allow explicitly providing a trusted_root.json file instead of relying on either TUF client. I'd be happy to chat more about this on a call.

@steiza
Copy link
Member

steiza commented Sep 25, 2024

I think we're still at odds a bit in terms of approach (but it's also possible I'm missing something!)

Ultimately, we have to decide what verification path we use in different scenarios, sigstore-go or pkg/cosign/verify.

Here's my understanding of the current state of things (which we can of course change):

Scenario Verification Path
Verifying with a new protocol buffer bundle (which requires a trusted root) sigstore-go (landed in #3796)
Verifying disparate signed materials or an old bundle with a trusted root sigstore-go (proposed in #3854)
Verifying anything with TUF v2 pkg/cosign/verify (proposed here in #3844)
Verifying anything with TUF v2 sigstore-go (suggested by @steiza - fetch the trusted root and then use proposed code in #3854)

It is certainly possible to use pkg/cosign/verify here with TUF v2, but I think our current tests are missing the case where there are several sets of verification material and today pkg/cosign/verify picks one to use instead of checking all of them, including at least SCT verification, VerifySet, and TSA verification that I listed out on #3854 (comment).

We could make this pull request address those cases. But ultimately, we need #3844 and #3854 to agree on what the verification path should be when you're using TUF v2 - sigstore-go or pkg/cosign/verify.

@steiza
Copy link
Member

steiza commented Sep 26, 2024

I was confused again this morning, and worried I lumped together too many use-cases, so I made a more detailed table:

Signed materials Verification materials Verification materials provided by Verification Path
Protocol buffer bundle requires trusted root trusted root file on disk sigstore-go (landed in #3796)
Protocol buffer bundle requires trusted root TUFv1 N/A - we require a trusted root, which TUFv1 does not provide
Protocol buffer bundle requires trusted root TUFv2 sigstore-go (suggested by @steiza - fetch the trusted root and then use code landed in #3796)
Not protocol buffer bundle trusted root trusted root file on disk sigstore-go (proposed in #3854)
Not protocol buffer bundle trusted root TUFv1 N/A - TUFv1 does not supply a trusted root, see "not trusted root" below
Not protocol buffer bundle trusted root TUFv2 pkg/cosign/verify (proposed here in #3844) or row below
Not protocol buffer bundle trusted root TUFv2 sigstore-go (suggested by @steiza - fetch the trusted root and then use proposed code in #3854) or row above
Not protocol buffer bundle not trusted root (e.g. disparate material via flags or ENV vars) TUFv1 pkg/cosign/verify (already implemented)
Not protocol buffer bundle not trusted root (e.g. disparate material via flags or ENV vars) TUFv2 This doesn't make sense - if you're using TUFv2 you're going to have a trusted root

I think I ended up in the same place though.

In the case where we don't have a new protocol buffer bundle, and we have a trusted root that we fetched with TUFv2, we need to decide if the verification path is going to be:

  • pkg/cosign/verify (in which case we need to audit and improve that verification path to handle the lists of possible verification material the trusted root provides) or
  • sigstore-go where we've already implemented that functionality (which means modifying commands individually)

@jku
Copy link
Member

jku commented Sep 26, 2024

thanks for writing that up zach, I'm trying to catch up and this is useful

@cmurphy
Copy link
Contributor Author

cmurphy commented Sep 26, 2024

@steiza thanks for the summary, I have a couple of minor clarifying questions:

This doesn't make sense - if you're using TUFv2 you're going to have a trusted root

Do you mean ideally, or currently? Using the TUF v2 client doesn't guarantee you've created a trusted_root.json and added it to your TUF repository.

pkg/cosign/verify (proposed here in #3844) or row below

Technically this PR does very little in pkg/cosign/verify.go. The scope of the TUF change also needs to include getting keys for the CT server which are needed during signing.

@jku
Copy link
Member

jku commented Sep 27, 2024

if new environment variables TUF_MIRROR, TUF_ROOT_JSON or
TUF_USE_TRUSTED_ROOT are set, use those to instantiate a TUF v2 client.

How do we expect these to get set? By end users?

It feels wrong if users need to set an environment variable to make the software use the recommended trust root mechanism.

@steiza
Copy link
Member

steiza commented Sep 27, 2024

Do you mean ideally, or currently? Using the TUF v2 client doesn't guarantee you've created a trusted_root.json and added it to your TUF repository.

Aha, I think I finally understand our different perspectives! I thought that when we use a TUF v2 client we were only going to fetch the trusted_root.json and not fetch individual files over TUF like ctfe.pub - but maybe that's that not the case? The way #3548 is worded made me think that we're doing the TUF v2 client upgrade and moving to using trusted_root.json at the same time.

If we are allowing fetching files like ctfe.pub over TUF v2, then I agree that we should use the contents of those files with the existing pkg/cosign/verify verification path (or signing, like you linked).

But if we're fetching trusted_root.json over TUF v2, that's when we need to decide if we're going to use sigstore-go or if we're going to make additional changes to pkg/cosign/verify in this pull request. The trusted root provides lists of possible verification material, which the pkg/cosign/verify was not originally written to handle.

For example, we verify the certificate with intermediateCerts *x509.CertPool that includes all the intermediate certs listed in the trusted root (great), but we always pick the first chain to verify the SCT (not great). I listed out two other similar cases on #3854 (comment), but I haven't done a close audit of pkg/cosign/verify either.

@cmurphy
Copy link
Contributor Author

cmurphy commented Feb 11, 2025

Done:

  • Made several fixes for bugs that the conformance tests caught and updated the e2e tests to avoid certain bugs slipping through
  • Updated the warning message to pass through the error from TUF
  • Rebased and fixed errors that arose from conflicts with Cody's latest change

Blocked:

@cmurphy cmurphy force-pushed the upgrade-tuf branch 2 times, most recently from 38bee43 to bbe5f24 Compare March 14, 2025 21:20
@cmurphy cmurphy force-pushed the upgrade-tuf branch 3 times, most recently from 114b8d2 to 00192be Compare May 5, 2025 20:46
@cmurphy
Copy link
Contributor Author

cmurphy commented May 5, 2025

This is passing tests now and should be ready for review. Apologies for the delay.

Comment on lines 77 to 79
trustedMaterial, err := cosign.TrustedRoot()
if err != nil {
ui.Warnf(context.Background(), "Could not fetch trusted_root.json from the TUF repository. Continuing with individual targets. Error from TUF: %v", err)
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still working my way through this PR, but I'm curious if this behavior is consistent with the existing code. I noticed that each signing/verification command starts by loading the Trusted Root from TUF, but if the user supplies their own Trusted Root JSON, I would expect that we wouldn't issue a TUF update (both for air-gapped mode and for speed). Do we currently always fetch TUF updates even if the user doesn't need it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++, although I think this comment makes more sense on the verify commands where we have --offline as a strong single if we're doing air-gapped verification.

You could be doing attest/sign commands in a restricted network environment (maybe there's a HSM attached to the network) although I'm not sure we have a flag we can key off of (like --offline for verify commands). I don't know enough about cosign usage to know if we need to support signing in restricted network environments.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an interesting point, because the sign commands don't have a --trusted-root flag. The trusted root is needed for signing because it does SCT verification as part of the fulcio workflow, and cosign has always been looking in the cached TUF root for this CT public key. If you try to verify a bundle and pass in a different trusted root with --trusted-root, the SCT check will fail. The sign commands need to have a --trusted-root flag if we want verification to work in general and especially without a new TUF fetch.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left the sign commands as-is because there is no flag to indicate where to get an offline trusted root. There should be, but I think that's outside of the scope of this already very large PR. I did rearrange the verify commands to make sure they don't try to fetch the TUF root online if a trusted root file is already provided.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine as-is for signing, and like you say we can add a flag if people need offline signing in another PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If --tlog-upload and --issue-certificate are false and --key is used, then there should be no need for TUF metadata.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--tlog-upload actually isn't relevant because we don't verify the tlog checkpoint during signing (yet). attest and attest-blob don't use --issue-certificate, so I used --key for those and both --key and --issue-certificate for sign and sign-blob.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good call on not needing tlog-upload. This SGTM!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this added yet, I don't see these changes in the PR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, forgot to push them, they're there now.

Copy link
Member

@steiza steiza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a monster pull request, although I did read it all the way through once (and then went back to several parts). Generally I think this is spot-on, but a few details I still have questions on.

Comment on lines 77 to 79
trustedMaterial, err := cosign.TrustedRoot()
if err != nil {
ui.Warnf(context.Background(), "Could not fetch trusted_root.json from the TUF repository. Continuing with individual targets. Error from TUF: %v", err)
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++, although I think this comment makes more sense on the verify commands where we have --offline as a strong single if we're doing air-gapped verification.

You could be doing attest/sign commands in a restricted network environment (maybe there's a HSM attached to the network) although I'm not sure we have a flag we can key off of (like --offline for verify commands). I don't know enough about cosign usage to know if we need to support signing in restricted network environments.

Copy link
Member

@steiza steiza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think once we fix the linter and e2e tests, this is good-to-go!

Comment on lines 77 to 79
trustedMaterial, err := cosign.TrustedRoot()
if err != nil {
ui.Warnf(context.Background(), "Could not fetch trusted_root.json from the TUF repository. Continuing with individual targets. Error from TUF: %v", err)
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine as-is for signing, and like you say we can add a flag if people need offline signing in another PR.

@cmurphy
Copy link
Contributor Author

cmurphy commented May 30, 2025

#4216 for the e2e test failures.

steiza
steiza previously approved these changes May 30, 2025
Copy link
Member

@steiza steiza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

Comment on lines 77 to 79
trustedMaterial, err := cosign.TrustedRoot()
if err != nil {
ui.Warnf(context.Background(), "Could not fetch trusted_root.json from the TUF repository. Continuing with individual targets. Error from TUF: %v", err)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens now when you sign with a provided key rather than using an identity/fetching a certificate? Unless you're initializing a signer using Fulcio I'm fairly certain that there is no attempt to fetch TUF targets in the background. (If there is, that's arguably a bug in Cosign).

With just a key and signing a blob, signing should be offline (signing is not offline of course if writing to an OCI registry or if --issue-certificate is set to fetch a fulcio cert). It's also worth noting that Cosign may be used behind a firewall and even if signing requires a network connection (signing/attesting a container with a key), there may be a limited set of allowed domains for outbound traffic. This isn't hypothetical either, we've gotten bug reports/FRs about this in the past.

Is there a set of flag values we can use to determine if TUF targets are needed?

For Cosign v3, we do need to clarify what --offline does to actually mean "there are no outbound connections" and disallow certain flag combos.

@haydentherapper
Copy link
Contributor

Just need to resolve the conflict, this looks great!!

@cmurphy cmurphy force-pushed the upgrade-tuf branch 2 times, most recently from c064331 to 3672ac0 Compare June 10, 2025 17:45
Use sigstore-go's TUF client to fetch the trusted_root.json from the TUF
mirror, if available. Where possible, use sigstore-go's verifiers which
natively accept the trusted root as its trusted material. Where there is
no trusted root available in TUF or sigstore-go doesn't support a use
case, fall back to the sigstore/sigstore TUF v1 client and the existing
verifiers in cosign.

Signed-off-by: Colleen Murphy <[email protected]>
Copy link
Contributor

@haydentherapper haydentherapper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing work on this!

@cmurphy cmurphy merged commit 32a2d62 into sigstore:main Jun 10, 2025
29 of 32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upgrade to latest Sigstore TUF client
7 participants