Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

building image from source run into invalid peer certificate: UnknownIssuer issue #1363

Open
kranurag7 opened this issue Nov 13, 2024 · 3 comments
Open

building image from source run into invalid peer certificate: UnknownIssuer issue #1363

kranurag7 opened this issue Nov 13, 2024 · 3 comments

Comments

@kranurag7
Copy link

I'm trying to build the image from source using wolfi toolings.

In istio, we already have the base image (https://github.com/istio/istio/blob/master/docker/iptables.yaml) and then we put the built ztunnel binary on top of that.

following are the apk packages that we pull in distroless variant of the image and I copied the same in ztunnel package as well.

$ syft istio/ztunnel:1.24.0-distroless
 ✔ Loaded image                                                                                                                                              istio/ztunnel:1.24.0-distroless
 ✔ Parsed image                                                                                                      sha256:52c18372cb344fb7c9c07cc3117db1cbcbe28592fff116b47fee291a41f66d1c
 ✔ Cataloged contents                                                                                                       a859dea8dd51b56f4a97af72010499fac182f0e3d1576b40d3512a06e04369a1
   ├── ✔ Packages                        [12 packages]
   ├── ✔ File digests                    [192 files]
   ├── ✔ File metadata                   [192 locations]
   └── ✔ Executables                     [146 executables]
NAME                    VERSION       TYPE
ca-certificates-bundle  20240705-r0   apk
glibc                   2.40-r2       apk
glibc-locale-posix      2.40-r2       apk
ip6tables               1.8.10-r4     apk
iptables                1.8.10-r4     apk
ld-linux                2.40-r2       apk
libgcc                  14.2.0-r3     apk
libmnl                  1.0.5-r4      apk
libnetfilter_conntrack  1.0.9-r4      apk
libnfnetlink            1.0.2-r4      apk
libnftnl                1.2.7-r0      apk
wolfi-baselayout        20230201-r15  apk

I've tried to include all these dependencies at runtime and build a package out of it (wolfi-dev/os#34028)

I was able to build the image using it but after installing the image following helm instructions, I'm running into following issues.

the main highlight logs to look into is following:

2024-11-13T11:47:57.726255Z    debug    hyper_util::client::legacy::connect::dns:xds{id=110}:resolve{host=istiod.istio-system.svc}    resolving    host="istiod.istio-system.svc"
2024-11-13T11:47:57.727082Z    debug    hyper_util::client::legacy::connect::http:xds{id=110}    connecting to 10.96.92.78:15012
2024-11-13T11:47:57.727186Z    debug    hyper_util::client::legacy::connect::http:xds{id=110}    connected to 10.96.92.78:15012
2024-11-13T11:47:57.727197Z    debug    rustls::client::hs:xds{id=110}    No cached session for DnsName("istiod.istio-system.svc")
2024-11-13T11:47:57.727258Z    debug    rustls::client::hs:xds{id=110}    Not resuming any session
2024-11-13T11:47:57.728934Z    debug    rustls::client::hs:xds{id=110}    Using ciphersuite TLS13_AES_128_GCM_SHA256
2024-11-13T11:47:57.728948Z    debug    rustls::client::tls13:xds{id=110}    Not resuming
2024-11-13T11:47:57.729018Z    debug    rustls::client::tls13:xds{id=110}    TLS1.3 encrypted extensions: [Protocols([ProtocolName(6832)])]
2024-11-13T11:47:57.729025Z    debug    rustls::client::hs:xds{id=110}    ALPN protocol is Some(b"h2")
2024-11-13T11:47:57.729032Z    debug    rustls::client::tls13:xds{id=110}    Got CertificateRequest CertificateRequestPayloadTls13 { context: , extensions: [Unknown(UnknownExtension { typ: S
tatusRequest, payload:  }), Unknown(UnknownExtension { typ: SCT, payload:  }), SignatureAlgorithms([RSA_PSS_SHA256, ECDSA_NISTP256_SHA256, ED25519, RSA_PSS_SHA384, RSA_PSS_SHA512, RSA_PKCS1_
HA256, RSA_PKCS1_SHA384, RSA_PKCS1_SHA512, ECDSA_NISTP384_SHA384, ECDSA_NISTP521_SHA512, RSA_PKCS1_SHA1, ECDSA_SHA1_Legacy[]), AuthorityNames([DistinguishedName(301831163014060355040a130d636
c75737465722e6c6f63616c)])] }
2024-11-13T11:47:57.729039Z    debug    rustls::client::common:xds{id=110}    Client auth requested but no cert/sigscheme available
2024-11-13T11:47:57.729136Z    warn    xds::client:xds{id=110}    XDS client connection error: gRPC connection error connecting to https://istiod.istio-system.svc:15012: status: Unknown, mes
sage: "client error (Connect)", source: invalid peer certificate: UnknownIssuer, retrying in 15s

From my understanding, it's connecting and then failing to recognize the certificate.

Environment details:

  • kind cluster on a x86 linux machine

Things I've tried are:

  • searching for similar issues on the repo, I found out we can update the env to fix this kubectl set env -n istio-system deploy/istiod ISTIOD_CUSTOM_HOST=localhost this didn't worked for me.
  • The good thing is that, when I replace the image back to istio/ztunnel then everything start working meaning I'm doing something wrong. Maybe I need to pass the tls flags during build time. As of now, I'm only doing cargo build --release
  • some issue mentioned that the istiod service should be running and I validated that it's running.
$ kubectl get svc -n istio-system
NAME     TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                                 AGE
istiod   ClusterIP   10.96.92.78   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP   57m

I'll appreciate more guidance here on how to build the image from source and if missed something in the deployment config. Thank you!!
@howardjohn
Copy link
Member

That is pretty odd. When we build the release it's just cargo build --release.

There is supposed to be a CA cert for Istiod loaded as a configmap. I'm not really sure how that could behave differently by just building the image yourself unless the source code has changes?

@kranurag7
Copy link
Author

I'm not really sure how that could behave differently by just building the image yourself unless the source code has changes?

source code has zero changes, we are just building using cargo build --release and putting all this together like upstream matching the dependencies one to one and we are building 1.24.0 tag as well.

This is the log I get when using upstream image:

2024-11-13T14:55:39.856741Z    info    hyper_util    listener established    address=[::]:15021 component="readiness"
2024-11-13T14:55:39.856784Z    info    app    shared proxy mode - in-pod mode enabled
2024-11-13T14:55:39.856850Z    info    hyper_util    listener established    address=127.0.0.1:15000 component="admin"
2024-11-13T14:55:39.856872Z    info    hyper_util    listener established    address=[::]:15020 component="stats"
2024-11-13T14:55:39.856877Z    info    readiness    Task 'dns proxy' complete (1.49516ms), still awaiting 3 tasks
2024-11-13T14:55:39.856880Z    info    readiness    Task 'proxy' complete (1.498978ms), still awaiting 2 tasks
2024-11-13T14:55:39.864478Z    info    xds::client:xds{id=1}    Stream established
2024-11-13T14:55:39.864517Z    info    xds::client:xds{id=1}    received response    type_url="type.googleapis.com/istio.workload.Address" size=15 removes=0
2024-11-13T14:55:39.864621Z    info    xds::client:xds{id=1}    received response    type_url="type.googleapis.com/istio.security.Authorization" size=0 removes=0
2024-11-13T14:55:39.864659Z    info    readiness    Task 'state manager' complete (9.277279ms), still awaiting 1 tasks
2024-11-13T14:55:39.864737Z    info    inpod::workloadmanager    handling new stream
2024-11-13T14:55:39.865281Z    info    inpod::statemanager    received snapshot sent
2024-11-13T14:55:39.865300Z    info    readiness    Task 'workload proxy manager' complete (9.917704ms), marking server ready
2024-11-13T14:55:40.343044Z    info    xds::client:xds{id=1}    received response    type_url="type.googleapis.com/istio.workload.Address" size=16 removes=0
2024-11-13T14:55:40.343169Z    info    xds::client:xds{id=1}    received response    type_url="type.googleapis.com/istio.security.Authorization" size=0 removes=0
2024-11-13T14:55:40.443506Z    info    xds::client:xds{id=1}    received response    type_url="type.googleapis.com/istio.workload.Address" size=1 removes=0

When I set image to the one I built locally it runs into the following issue documented above:

2024-11-13T15:04:44.863593Z    info    hyper_util    listener established    address=[::]:15021 component="readiness"
2024-11-13T15:04:44.863628Z    info    app    shared proxy mode - in-pod mode enabled
2024-11-13T15:04:44.863677Z    info    hyper_util    listener established    address=127.0.0.1:15000 component="admin"
2024-11-13T15:04:44.863698Z    info    hyper_util    listener established    address=[::]:15020 component="stats"
2024-11-13T15:04:44.863703Z    info    readiness    Task 'dns proxy' complete (1.047833ms), still awaiting 3 tasks
2024-11-13T15:04:44.863705Z    info    readiness    Task 'proxy' complete (1.050806ms), still awaiting 2 tasks
2024-11-13T15:04:44.867381Z    warn    xds::client:xds{id=1}    XDS client connection error: gRPC connection error connecting to https://istiod.istio-system.svc:15012: status: Unknown, messa
ge: "client error (Connect)", source: invalid peer certificate: UnknownIssuer, retrying in 20ms
2024-11-13T15:04:44.891905Z    warn    xds::client:xds{id=2}    XDS client connection error: gRPC connection error connecting to https://istiod.istio-system.svc:15012: status: Unknown, messa
ge: "client error (Connect)", source: invalid peer certificate: UnknownIssuer, retrying in 40ms
2024-11-13T15:04:44.936608Z    warn    xds::client:xds{id=3}    XDS client connection error: gRPC connection error connecting to https://istiod.istio-system.svc:15012: status: Unknown, messa
ge: "client error (Connect)", source: invalid peer certificate: UnknownIssuer, retrying in 80ms
2024-11-13T15:04:45.021163Z    warn    xds::client:xds{id=4}    XDS client connection error: gRPC connection error connecting to https://istiod.istio-system.svc:15012: status: Unknown, messa
ge: "client error (Connect)", source: invalid peer certificate: UnknownIssuer, retrying in 160ms

In the cluster, we have the following configmaps as of now.

 $ kubectl get cm -A | grep istio
default              istio-ca-root-cert                                     1      16m
istio-system         istio                                                  2      16m
istio-system         istio-ca-root-cert                                     1      16m
istio-system         istio-cni-config                                       13     16m
istio-system         istio-gateway-status-leader                            0      16m
istio-system         istio-leader                                           0      16m
istio-system         istio-namespace-controller-election                    0      16m
istio-system         istio-sidecar-injector                                 2      16m
istio-system         istio-status-leader                                    0      16m
istio-system         kube-root-ca.crt                                       1      16m
kube-system          istio-ca-root-cert                                     1      16m

The controller is revolving around this place.

024-11-13T15:13:44.180824Z    debug    hyper_util::client::legacy::connect::http:xds{id=12}    connecting to 10.96.141.126:15012
2024-11-13T15:13:44.180939Z    debug    hyper_util::client::legacy::connect::http:xds{id=12}    connected to 10.96.141.126:15012
2024-11-13T15:13:44.180956Z    debug    rustls::client::hs:xds{id=12}    No cached session for DnsName("istiod.istio-system.svc")
2024-11-13T15:13:44.181022Z    debug    rustls::client::hs:xds{id=12}    Not resuming any session
2024-11-13T15:13:44.182810Z    debug    rustls::client::hs:xds{id=12}    Using ciphersuite TLS13_AES_128_GCM_SHA256
2024-11-13T15:13:44.182823Z    debug    rustls::client::tls13:xds{id=12}    Not resuming
2024-11-13T15:13:44.182887Z    debug    rustls::client::tls13:xds{id=12}    TLS1.3 encrypted extensions: [Protocols([ProtocolName(6832)])]
2024-11-13T15:13:44.182894Z    debug    rustls::client::hs:xds{id=12}    ALPN protocol is Some(b"h2")
2024-11-13T15:13:44.182902Z    debug    rustls::client::tls13:xds{id=12}    Got CertificateRequest CertificateRequestPayloadTls13 { context: , extensions: [Unknown(UnknownExtension { typ: St
atusRequest, payload:  }), Unknown(UnknownExtension { typ: SCT, payload:  }), SignatureAlgorithms([RSA_PSS_SHA256, ECDSA_NISTP256_SHA256, ED25519, RSA_PSS_SHA384, RSA_PSS_SHA512, RSA_PKCS1_S
A256, RSA_PKCS1_SHA384, RSA_PKCS1_SHA512, ECDSA_NISTP384_SHA384, ECDSA_NISTP521_SHA512, RSA_PKCS1_SHA1, ECDSA_SHA1_Legacy[]), AuthorityNames([DistinguishedName(301831163014060355040a130d636c
75737465722e6c6f63616c)])] }
2024-11-13T15:13:44.182909Z    debug    rustls::client::common:xds{id=12}    Client auth requested but no cert/sigscheme available
2024-11-13T15:13:44.183004Z    warn    xds::client:xds{id=12}    XDS client connection error: gRPC connection error connecting to https://istiod.istio-system.svc:15012: status: Unknown, mess
age: "client error (Connect)", source: invalid peer certificate: UnknownIssuer, retrying in 15s

it's connecting, and then not resuming and then finally running into error.

Please let me know I can provide more details around it and I'm happy to provide any. Thanks so much for looking into it while you're at kubecon. Have fun time there. :)

@howardjohn
Copy link
Member

howardjohn commented Nov 13, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@howardjohn @kranurag7