building image from source run into invalid peer certificate: UnknownIssuer issue

[kranurag7]

I'm trying to build the image from source using wolfi toolings.

In istio, we already have the base image (https://github.com/istio/istio/blob/master/docker/iptables.yaml) and then we put the built ztunnel binary on top of that.

following are the apk packages that we pull in distroless variant of the image and I copied the same in ztunnel package as well.

$ syft istio/ztunnel:1.24.0-distroless
 ✔ Loaded image                                                                                                                                              istio/ztunnel:1.24.0-distroless
 ✔ Parsed image                                                                                                      sha256:52c18372cb344fb7c9c07cc3117db1cbcbe28592fff116b47fee291a41f66d1c
 ✔ Cataloged contents                                                                                                       a859dea8dd51b56f4a97af72010499fac182f0e3d1576b40d3512a06e04369a1
   ├── ✔ Packages                        [12 packages]
   ├── ✔ File digests                    [192 files]
   ├── ✔ File metadata                   [192 locations]
   └── ✔ Executables                     [146 executables]
NAME                    VERSION       TYPE
ca-certificates-bundle  20240705-r0   apk
glibc                   2.40-r2       apk
glibc-locale-posix      2.40-r2       apk
ip6tables               1.8.10-r4     apk
iptables                1.8.10-r4     apk
ld-linux                2.40-r2       apk
libgcc                  14.2.0-r3     apk
libmnl                  1.0.5-r4      apk
libnetfilter_conntrack  1.0.9-r4      apk
libnfnetlink            1.0.2-r4      apk
libnftnl                1.2.7-r0      apk
wolfi-baselayout        20230201-r15  apk

I've tried to include all these dependencies at runtime and build a package out of it (wolfi-dev/os#34028)

I was able to build the image using it but after installing the image following helm instructions, I'm running into following issues.

the main highlight logs to look into is following:

2024-11-13T11:47:57.726255Z    debug    hyper_util::client::legacy::connect::dns:xds{id=110}:resolve{host=istiod.istio-system.svc}    resolving    host="istiod.istio-system.svc"
2024-11-13T11:47:57.727082Z    debug    hyper_util::client::legacy::connect::http:xds{id=110}    connecting to 10.96.92.78:15012
2024-11-13T11:47:57.727186Z    debug    hyper_util::client::legacy::connect::http:xds{id=110}    connected to 10.96.92.78:15012
2024-11-13T11:47:57.727197Z    debug    rustls::client::hs:xds{id=110}    No cached session for DnsName("istiod.istio-system.svc")
2024-11-13T11:47:57.727258Z    debug    rustls::client::hs:xds{id=110}    Not resuming any session
2024-11-13T11:47:57.728934Z    debug    rustls::client::hs:xds{id=110}    Using ciphersuite TLS13_AES_128_GCM_SHA256
2024-11-13T11:47:57.728948Z    debug    rustls::client::tls13:xds{id=110}    Not resuming
2024-11-13T11:47:57.729018Z    debug    rustls::client::tls13:xds{id=110}    TLS1.3 encrypted extensions: [Protocols([ProtocolName(6832)])]
2024-11-13T11:47:57.729025Z    debug    rustls::client::hs:xds{id=110}    ALPN protocol is Some(b"h2")
2024-11-13T11:47:57.729032Z    debug    rustls::client::tls13:xds{id=110}    Got CertificateRequest CertificateRequestPayloadTls13 { context: , extensions: [Unknown(UnknownExtension { typ: S
tatusRequest, payload:  }), Unknown(UnknownExtension { typ: SCT, payload:  }), SignatureAlgorithms([RSA_PSS_SHA256, ECDSA_NISTP256_SHA256, ED25519, RSA_PSS_SHA384, RSA_PSS_SHA512, RSA_PKCS1_
HA256, RSA_PKCS1_SHA384, RSA_PKCS1_SHA512, ECDSA_NISTP384_SHA384, ECDSA_NISTP521_SHA512, RSA_PKCS1_SHA1, ECDSA_SHA1_Legacy[]), AuthorityNames([DistinguishedName(301831163014060355040a130d636
c75737465722e6c6f63616c)])] }
2024-11-13T11:47:57.729039Z    debug    rustls::client::common:xds{id=110}    Client auth requested but no cert/sigscheme available
2024-11-13T11:47:57.729136Z    warn    xds::client:xds{id=110}    XDS client connection error: gRPC connection error connecting to https://istiod.istio-system.svc:15012: status: Unknown, mes
sage: "client error (Connect)", source: invalid peer certificate: UnknownIssuer, retrying in 15s

From my understanding, it's connecting and then failing to recognize the certificate.

Environment details:

• kind cluster on a x86 linux machine

Things I've tried are:

• searching for similar issues on the repo, I found out we can update the env to fix this kubectl set env -n istio-system deploy/istiod ISTIOD_CUSTOM_HOST=localhost this didn't worked for me.
• The good thing is that, when I replace the image back to istio/ztunnel then everything start working meaning I'm doing something wrong. Maybe I need to pass the tls flags during build time. As of now, I'm only doing cargo build --release
• some issue mentioned that the istiod service should be running and I validated that it's running.

$ kubectl get svc -n istio-system
NAME     TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                                 AGE
istiod   ClusterIP   10.96.92.78   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP   57m

I'll appreciate more guidance here on how to build the image from source and if missed something in the deployment config. Thank you!!

[howardjohn]

That is pretty odd. When we build the release it's just cargo build --release.

There is supposed to be a CA cert for Istiod loaded as a configmap. I'm not really sure how that could behave differently by just building the image yourself unless the source code has changes?

[kranurag7]

I'm not really sure how that could behave differently by just building the image yourself unless the source code has changes?

source code has zero changes, we are just building using cargo build --release and putting all this together like upstream matching the dependencies one to one and we are building 1.24.0 tag as well.

This is the log I get when using upstream image:

2024-11-13T14:55:39.856741Z    info    hyper_util    listener established    address=[::]:15021 component="readiness"
2024-11-13T14:55:39.856784Z    info    app    shared proxy mode - in-pod mode enabled
2024-11-13T14:55:39.856850Z    info    hyper_util    listener established    address=127.0.0.1:15000 component="admin"
2024-11-13T14:55:39.856872Z    info    hyper_util    listener established    address=[::]:15020 component="stats"
2024-11-13T14:55:39.856877Z    info    readiness    Task 'dns proxy' complete (1.49516ms), still awaiting 3 tasks
2024-11-13T14:55:39.856880Z    info    readiness    Task 'proxy' complete (1.498978ms), still awaiting 2 tasks
2024-11-13T14:55:39.864478Z    info    xds::client:xds{id=1}    Stream established
2024-11-13T14:55:39.864517Z    info    xds::client:xds{id=1}    received response    type_url="type.googleapis.com/istio.workload.Address" size=15 removes=0
2024-11-13T14:55:39.864621Z    info    xds::client:xds{id=1}    received response    type_url="type.googleapis.com/istio.security.Authorization" size=0 removes=0
2024-11-13T14:55:39.864659Z    info    readiness    Task 'state manager' complete (9.277279ms), still awaiting 1 tasks
2024-11-13T14:55:39.864737Z    info    inpod::workloadmanager    handling new stream
2024-11-13T14:55:39.865281Z    info    inpod::statemanager    received snapshot sent
2024-11-13T14:55:39.865300Z    info    readiness    Task 'workload proxy manager' complete (9.917704ms), marking server ready
2024-11-13T14:55:40.343044Z    info    xds::client:xds{id=1}    received response    type_url="type.googleapis.com/istio.workload.Address" size=16 removes=0
2024-11-13T14:55:40.343169Z    info    xds::client:xds{id=1}    received response    type_url="type.googleapis.com/istio.security.Authorization" size=0 removes=0
2024-11-13T14:55:40.443506Z    info    xds::client:xds{id=1}    received response    type_url="type.googleapis.com/istio.workload.Address" size=1 removes=0

When I set image to the one I built locally it runs into the following issue documented above:

2024-11-13T15:04:44.863593Z    info    hyper_util    listener established    address=[::]:15021 component="readiness"
2024-11-13T15:04:44.863628Z    info    app    shared proxy mode - in-pod mode enabled
2024-11-13T15:04:44.863677Z    info    hyper_util    listener established    address=127.0.0.1:15000 component="admin"
2024-11-13T15:04:44.863698Z    info    hyper_util    listener established    address=[::]:15020 component="stats"
2024-11-13T15:04:44.863703Z    info    readiness    Task 'dns proxy' complete (1.047833ms), still awaiting 3 tasks
2024-11-13T15:04:44.863705Z    info    readiness    Task 'proxy' complete (1.050806ms), still awaiting 2 tasks
2024-11-13T15:04:44.867381Z    warn    xds::client:xds{id=1}    XDS client connection error: gRPC connection error connecting to https://istiod.istio-system.svc:15012: status: Unknown, messa
ge: "client error (Connect)", source: invalid peer certificate: UnknownIssuer, retrying in 20ms
2024-11-13T15:04:44.891905Z    warn    xds::client:xds{id=2}    XDS client connection error: gRPC connection error connecting to https://istiod.istio-system.svc:15012: status: Unknown, messa
ge: "client error (Connect)", source: invalid peer certificate: UnknownIssuer, retrying in 40ms
2024-11-13T15:04:44.936608Z    warn    xds::client:xds{id=3}    XDS client connection error: gRPC connection error connecting to https://istiod.istio-system.svc:15012: status: Unknown, messa
ge: "client error (Connect)", source: invalid peer certificate: UnknownIssuer, retrying in 80ms
2024-11-13T15:04:45.021163Z    warn    xds::client:xds{id=4}    XDS client connection error: gRPC connection error connecting to https://istiod.istio-system.svc:15012: status: Unknown, messa
ge: "client error (Connect)", source: invalid peer certificate: UnknownIssuer, retrying in 160ms

In the cluster, we have the following configmaps as of now.

 $ kubectl get cm -A | grep istio
default              istio-ca-root-cert                                     1      16m
istio-system         istio                                                  2      16m
istio-system         istio-ca-root-cert                                     1      16m
istio-system         istio-cni-config                                       13     16m
istio-system         istio-gateway-status-leader                            0      16m
istio-system         istio-leader                                           0      16m
istio-system         istio-namespace-controller-election                    0      16m
istio-system         istio-sidecar-injector                                 2      16m
istio-system         istio-status-leader                                    0      16m
istio-system         kube-root-ca.crt                                       1      16m
kube-system          istio-ca-root-cert                                     1      16m

The controller is revolving around this place.

024-11-13T15:13:44.180824Z    debug    hyper_util::client::legacy::connect::http:xds{id=12}    connecting to 10.96.141.126:15012
2024-11-13T15:13:44.180939Z    debug    hyper_util::client::legacy::connect::http:xds{id=12}    connected to 10.96.141.126:15012
2024-11-13T15:13:44.180956Z    debug    rustls::client::hs:xds{id=12}    No cached session for DnsName("istiod.istio-system.svc")
2024-11-13T15:13:44.181022Z    debug    rustls::client::hs:xds{id=12}    Not resuming any session
2024-11-13T15:13:44.182810Z    debug    rustls::client::hs:xds{id=12}    Using ciphersuite TLS13_AES_128_GCM_SHA256
2024-11-13T15:13:44.182823Z    debug    rustls::client::tls13:xds{id=12}    Not resuming
2024-11-13T15:13:44.182887Z    debug    rustls::client::tls13:xds{id=12}    TLS1.3 encrypted extensions: [Protocols([ProtocolName(6832)])]
2024-11-13T15:13:44.182894Z    debug    rustls::client::hs:xds{id=12}    ALPN protocol is Some(b"h2")
2024-11-13T15:13:44.182902Z    debug    rustls::client::tls13:xds{id=12}    Got CertificateRequest CertificateRequestPayloadTls13 { context: , extensions: [Unknown(UnknownExtension { typ: St
atusRequest, payload:  }), Unknown(UnknownExtension { typ: SCT, payload:  }), SignatureAlgorithms([RSA_PSS_SHA256, ECDSA_NISTP256_SHA256, ED25519, RSA_PSS_SHA384, RSA_PSS_SHA512, RSA_PKCS1_S
A256, RSA_PKCS1_SHA384, RSA_PKCS1_SHA512, ECDSA_NISTP384_SHA384, ECDSA_NISTP521_SHA512, RSA_PKCS1_SHA1, ECDSA_SHA1_Legacy[]), AuthorityNames([DistinguishedName(301831163014060355040a130d636c
75737465722e6c6f63616c)])] }
2024-11-13T15:13:44.182909Z    debug    rustls::client::common:xds{id=12}    Client auth requested but no cert/sigscheme available
2024-11-13T15:13:44.183004Z    warn    xds::client:xds{id=12}    XDS client connection error: gRPC connection error connecting to https://istiod.istio-system.svc:15012: status: Unknown, mess
age: "client error (Connect)", source: invalid peer certificate: UnknownIssuer, retrying in 15s

it's connecting, and then not resuming and then finally running into error.

Please let me know I can provide more details around it and I'm happy to provide any. Thanks so much for looking into it while you're at kubecon. Have fun time there. :)

[howardjohn]

strange! One thing you could try to do to isolate things is to put the upstream binary into the wolfi image and/or the custom build into the non-wolfi image (although the upstream distroless is already using wolfi...). Also maybe double check in the config that it's actually loading the proper ca cert. It should print it out on the startup

…

On Wed, Nov 13, 2024, 8:16 AM kranurag7 ***@***.***> wrote: I'm not really sure how that could behave differently by just building the image yourself unless the source code has changes? source code has zero changes, we are just building using cargo build --release and putting all this together like upstream matching the dependencies one to one and we are building 1.24.0 tag as well. This is the log I get when using upstream image: 2024-11-13T14:55:39.856741Z info hyper_util listener established address=[::]:15021 component="readiness" 2024-11-13T14:55:39.856784Z info app shared proxy mode - in-pod mode enabled 2024-11-13T14:55:39.856850Z info hyper_util listener established address=127.0.0.1:15000 component="admin" 2024-11-13T14:55:39.856872Z info hyper_util listener established address=[::]:15020 component="stats" 2024-11-13T14:55:39.856877Z info readiness Task 'dns proxy' complete (1.49516ms), still awaiting 3 tasks 2024-11-13T14:55:39.856880Z info readiness Task 'proxy' complete (1.498978ms), still awaiting 2 tasks 2024-11-13T14:55:39.864478Z info xds::client:xds{id=1} Stream established 2024-11-13T14:55:39.864517Z info xds::client:xds{id=1} received response type_url="type.googleapis.com/istio.workload.Address" size=15 removes=0 2024-11-13T14:55:39.864621Z info xds::client:xds{id=1} received response type_url="type.googleapis.com/istio.security.Authorization" size=0 removes=0 2024-11-13T14:55:39.864659Z info readiness Task 'state manager' complete (9.277279ms), still awaiting 1 tasks 2024-11-13T14:55:39.864737Z info inpod::workloadmanager handling new stream 2024-11-13T14:55:39.865281Z info inpod::statemanager received snapshot sent 2024-11-13T14:55:39.865300Z info readiness Task 'workload proxy manager' complete (9.917704ms), marking server ready 2024-11-13T14:55:40.343044Z info xds::client:xds{id=1} received response type_url="type.googleapis.com/istio.workload.Address" size=16 removes=0 2024-11-13T14:55:40.343169Z info xds::client:xds{id=1} received response type_url="type.googleapis.com/istio.security.Authorization" size=0 removes=0 2024-11-13T14:55:40.443506Z info xds::client:xds{id=1} received response type_url="type.googleapis.com/istio.workload.Address" size=1 removes=0 When I set image to the one I built locally it runs into the following issue documented above: 2024-11-13T15:04:44.863593Z info hyper_util listener established address=[::]:15021 component="readiness" 2024-11-13T15:04:44.863628Z info app shared proxy mode - in-pod mode enabled 2024-11-13T15:04:44.863677Z info hyper_util listener established address=127.0.0.1:15000 component="admin" 2024-11-13T15:04:44.863698Z info hyper_util listener established address=[::]:15020 component="stats" 2024-11-13T15:04:44.863703Z info readiness Task 'dns proxy' complete (1.047833ms), still awaiting 3 tasks 2024-11-13T15:04:44.863705Z info readiness Task 'proxy' complete (1.050806ms), still awaiting 2 tasks 2024-11-13T15:04:44.867381Z warn xds::client:xds{id=1} XDS client connection error: gRPC connection error connecting to https://istiod.istio-system.svc:15012: status: Unknown, messa ge: "client error (Connect)", source: invalid peer certificate: UnknownIssuer, retrying in 20ms 2024-11-13T15:04:44.891905Z warn xds::client:xds{id=2} XDS client connection error: gRPC connection error connecting to https://istiod.istio-system.svc:15012: status: Unknown, messa ge: "client error (Connect)", source: invalid peer certificate: UnknownIssuer, retrying in 40ms 2024-11-13T15:04:44.936608Z warn xds::client:xds{id=3} XDS client connection error: gRPC connection error connecting to https://istiod.istio-system.svc:15012: status: Unknown, messa ge: "client error (Connect)", source: invalid peer certificate: UnknownIssuer, retrying in 80ms 2024-11-13T15:04:45.021163Z warn xds::client:xds{id=4} XDS client connection error: gRPC connection error connecting to https://istiod.istio-system.svc:15012: status: Unknown, messa ge: "client error (Connect)", source: invalid peer certificate: UnknownIssuer, retrying in 160ms In the cluster, we have the following configmaps as of now. $ kubectl get cm -A | grep istio default istio-ca-root-cert 1 16m istio-system istio 2 16m istio-system istio-ca-root-cert 1 16m istio-system istio-cni-config 13 16m istio-system istio-gateway-status-leader 0 16m istio-system istio-leader 0 16m istio-system istio-namespace-controller-election 0 16m istio-system istio-sidecar-injector 2 16m istio-system istio-status-leader 0 16m istio-system kube-root-ca.crt 1 16m kube-system istio-ca-root-cert 1 16m The controller is revolving around this place. 024-11-13T15:13:44.180824Z debug hyper_util::client::legacy::connect::http:xds{id=12} connecting to 10.96.141.126:15012 2024-11-13T15:13:44.180939Z debug hyper_util::client::legacy::connect::http:xds{id=12} connected to 10.96.141.126:15012 2024-11-13T15:13:44.180956Z debug rustls::client::hs:xds{id=12} No cached session for DnsName("istiod.istio-system.svc") 2024-11-13T15:13:44.181022Z debug rustls::client::hs:xds{id=12} Not resuming any session 2024-11-13T15:13:44.182810Z debug rustls::client::hs:xds{id=12} Using ciphersuite TLS13_AES_128_GCM_SHA256 2024-11-13T15:13:44.182823Z debug rustls::client::tls13:xds{id=12} Not resuming 2024-11-13T15:13:44.182887Z debug rustls::client::tls13:xds{id=12} TLS1.3 encrypted extensions: [Protocols([ProtocolName(6832)])] 2024-11-13T15:13:44.182894Z debug rustls::client::hs:xds{id=12} ALPN protocol is Some(b"h2") 2024-11-13T15:13:44.182902Z debug rustls::client::tls13:xds{id=12} Got CertificateRequest CertificateRequestPayloadTls13 { context: , extensions: [Unknown(UnknownExtension { typ: St atusRequest, payload: }), Unknown(UnknownExtension { typ: SCT, payload: }), SignatureAlgorithms([RSA_PSS_SHA256, ECDSA_NISTP256_SHA256, ED25519, RSA_PSS_SHA384, RSA_PSS_SHA512, RSA_PKCS1_S A256, RSA_PKCS1_SHA384, RSA_PKCS1_SHA512, ECDSA_NISTP384_SHA384, ECDSA_NISTP521_SHA512, RSA_PKCS1_SHA1, ECDSA_SHA1_Legacy[]), AuthorityNames([DistinguishedName(301831163014060355040a130d636c 75737465722e6c6f63616c)])] } 2024-11-13T15:13:44.182909Z debug rustls::client::common:xds{id=12} Client auth requested but no cert/sigscheme available 2024-11-13T15:13:44.183004Z warn xds::client:xds{id=12} XDS client connection error: gRPC connection error connecting to https://istiod.istio-system.svc:15012: status: Unknown, mess age: "client error (Connect)", source: invalid peer certificate: UnknownIssuer, retrying in 15s it's connecting, and then not resuming and then finally running into error. Please let me know I can provide more details around it and I'm happy to provide any. Thanks so much for looking into it while you're at kubecon. Have fun time there. :) — Reply to this email directly, view it on GitHub <#1363 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXNTZZP7U6EUGKWPCOT2ANUMFAVCNFSM6AAAAABRWJ3O56VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZTHEYTENJVG4> . You are receiving this because you commented.Message ID: ***@***.***>

[kranurag7]

Thanks for the Hint John.

The mistake I was committing was setting work-dir inside apko config. I was setting /home/nonroot looking at the base image that ztunnel uses but we overwrite it again while building ztunnel image and set it to /

This is working good now. I'll close this issue given this is resolved now.