Signature not found in GCP Artifact Registry

[david-freistrom]

What is not working as expected?

After notation sign a container image, GCP artifact registry shows the signature as an OCI artifact in GCP artifact-registry beside the docker image but no attachments are defined and verify or ls cannot find attached signature.

No attachments on the image nor on the signature OCI artifacts But the Artifacts exist and refers to the image in the manifest subject.

notation sign --signature-format jws --id personal/github/all --plugin hc-vault --plugin-config=kvName=kubeflow --plugin-config=transitKeyName=image-signing europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47

Successfully signed europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47

 notation ls europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47

europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47 has no associated signature

notation verify europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47

Error: signature verification failed: no signature is associated with "europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47", make sure the artifact was signed successfully

cat /home/david/.config/notation/trustpolicy.json

{
    "version": "1.0",
    "trustPolicies": [
        {
            "name": "example-images",
            "registryScopes": [ "*" ],
            "signatureVerification": {
                "level" : "strict"
            },
            "trustStores": [ "ca:example.com" ],
            "trustedIdentities": [
                "*"
            ]
        }
    ]
}

What did you expect to happen?

notation verify and ls shows me a valid signature attached to the image

How can we reproduce it?

Build and Push an docker image to artifact registry in GCP
sign it using notation
list or verify it using notation

Describe your environment

Linux Ubuntu 24.04.2 LTS

5.15.167.4-microsoft-standard-WSL2 #1 SMP Tue Nov 5 00:21:55 UTC 2024

Notation - a tool to sign and verify artifacts.
Version: 1.3.1
Go version: go1.23.6
Git commit: bb571dd

What is the version of your Notation CLI or Notation Library?

Notation - a tool to sign and verify artifacts.

Version: 1.3.1
Go version: go1.23.6
Git commit: bb571dd

[FeynmanZhou]

Hi @david-freistrom ,

In Notation v1.3.1, the notation sign command defaults to storing signatures using the OCI referrers tag schema for maximum compatibility. The notation verify/list/inspect commands will attempt to use the referrers API first and automatically fall back to the referrers tag schema if the referrers API is not supported by the registry. See this blog post for context.

It looks like the fallback is not executed as expected. Can you please try to add a flag --allow-referrers-api false to your notation list and notation verify commands as follows? This is an experimental flag in v1.x.

export NOTATION_EXPERIMENTAL=1

notation ls europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47 --allow-referrers-api false

notation verify europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47 --allow-referrers-api false

@david-freistrom After reading through your issue, one possible cause is that GCP Artifact Registry is not fully OCI 1.1 compliant.

[david-freistrom]

@Two-Hearts Small correction:
Attachments are present at the origin image. Didn't saw it in the first. But Notation doesn't work as expected for verify and ls

Regarding GCP Artifact Registry announcements, OCI 1.1 support was generally available at Oct. 03 2024.

[david-freistrom]

I am wondering why notation creates a second artifact with the digest as a tag beside the one with semantic version as a tag.
The attachments are only added to the semantic versioned artifact but verify looks only for the digest tagged artifact.
(It doesn't matter if I use semantic versioned tag or the digest for verify)

notation verify europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47 -v

INFO Allowed to access the referrers API, fallback if not supported

INFO Reference sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47 resolved to manifest descriptor: {MediaType:application/vnd.docker.distribution.manifest.v2+json Digest:sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47 Size:1159 URLs:[] Annotations:map[] Data:[] Platform:<nil> ArtifactType:}

INFO Checking whether signature verification should be skipped or not

INFO Trust policy configuration: &{Name:corporate-data-analytics-images RegistryScopes:[*] SignatureVerification:{VerificationLevel:strict Override:map[] VerifyTimestamp:} TrustStores:[ca:example.org] TrustedIdentities:[*]}
INFO Check over. The signature verification level is not set to 'skip' in the trust policy.

Error: signature verification failed: no signature is associated with "europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47", make sure the artifact was signed successfully

[david-freistrom]

Verify works with notation 1.1.2. Looks like a bug in 1.3.1.

notation-1.1.2 verify europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc:1.0 -v

INFO Using the referrers tag schema

INFO Reference 1.0 resolved to manifest descriptor: {MediaType:application/vnd.docker.distribution.manifest.v2+json Digest:sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47 Size:1159 URLs:[] Annotations:map[] Data:[] Platform:<nil> ArtifactType:}
Warning: Always verify the artifact using digest(@sha256:...) rather than a tag(:1.0) because resolved digest may not point to the same signed artifact, as tags are mutable.

INFO Checking whether signature verification should be skipped or not

INFO Trust policy configuration: &{Name:corporate-data-analytics-images RegistryScopes:[*] SignatureVerification:{VerificationLevel:strict Override:map[]} TrustStores:[ca:example.org] TrustedIdentities:[*]}

INFO Check over. Trust policy is not configured to skip signature verification

INFO Processing signature with manifest mediaType: application/vnd.oci.image.manifest.v1+json and digest: sha256:fc3d42c24a1742103ea57b687c990feec3eea2278bcece62a79a590018bb23d9

INFO Trust policy configuration: &{Name:corporate-data-analytics-images RegistryScopes:[*] SignatureVerification:{VerificationLevel:strict Override:map[]} TrustStores:[ca:example.org] TrustedIdentities:[*]}
Successfully verified signature for europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47

[david-freistrom]

@FeynmanZhou

export NOTATION_EXPERIMENTAL=1

Still not works. But notation 1.1.2 works as expected without your suggested settings.

@FeynmanZhou

export NOTATION_EXPERIMENTAL=1

Still not works. But notation 1.1.2 works as expected without your suggested settings.

@david-freistrom The above makes the problem even more likely to be a GCP side issue. Because with Notation 1.1.2, when sign/verify without export NOTATION_EXPERIMENTAL=1, Notation will always use referrers tag schema (OCI 1.0) instead of trying referrers api first, if not supported, then fallback to tag schema (OCI 1.1). Since GCP is OCI 1.0 compliant, everything works fine.

However, when using Notation v1.3.1, the verify/list/inspect commands automatically tries the referrers api first (OCI 1.1), your error starts to pop up. Because GCP does not fully support the try referrers api first then fall back to referrers tag schema (OCI 1.1) mechanism. If they do fully support it, the fall back should work, and the verification workflow eventually works the same as Notation v1.1.2.

(Note: Notation v1.3.1 has been tested with other OCI 1.1 compliant registries, a fully OCI 1.1 compliant registry won't return this error).