strimzi kafka operator using custom CA with nginx ingress controller #11627

charansaip · 2025-07-09T09:45:22Z

charansaip
Jul 9, 2025

We have deployed kafka strimzi operator : 0.44.0.

we deployed kafka cluster component with custom CA certs wit ngninx ingress controller.
kafka connector component
ERROR :
Failed authentication with kafka.bootstrap.sercer/LB IP ( ssl handshake failed)
Received fatal alert unrecognized name

im-konge · 2025-07-09T11:04:51Z

im-konge
Jul 9, 2025
Maintainer

Hey, this is really small info that you provided.
It would be great to share full logs and configuration of the Kafka cluster.

If you have failed authentication with custom certificates, it would indicate that you did something wrong when configuring it in Kafka.
Everything needed is mentioned in our documentation.

0 replies

charansaip · 2025-07-09T11:53:54Z

charansaip
Jul 9, 2025
Author

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: my-cluster
spec:
volumes:
- name: server-properties
configMap:
name: server.properties
kafka:
version: 3.5.0
replicas: 1
listeners:
- name: external
port: 9094
type: ingress
tls: true
authentication:
type: tls
configuration:
class: nginx
brokerCertChainAndKey:
secretName: kafka-wildcard-cert
certificate: tls.crt
key: tls.key
bootstrap:
host: dev.kafka.me.com
annotations:
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
brokers:
- broker: 0
host: devb0.kafka.me.com
annotations:
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
- broker: 1
host: devb1.kafka.me.com
annotations:
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
- broker: 2
host: devb2.kafka.me.com
annotations:
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
config:
offsets.topic.replication.factor: 1
transaction.state.log.replication.factor: 1
transaction.state.log.min.isr: 1
log.message.format.version: "2.7"
inter.broker.protocol.version: "2.7"
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 10Gi
deleteClaim: false
configMap:
name: client.properties
zookeeper:
replicas: 1
storage:
type: persistent-claim
size: 10Gi
deleteClaim: false
entityOperator:
topicOperator: {}
userOperator: {}

0 replies

charansaip · 2025-07-09T12:02:51Z

charansaip
Jul 9, 2025
Author

Basic configuration (required)

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect # (1)
metadata:
name: my-connect-cluster
annotations:
strimzi.io/use-connector-resources: "true" # (2)

Deployment specifications

spec:

Replicas (required)

replicas: 3 # (3)

Bootstrap servers (required)

bootstrapServers: dev.kafka.me.com:443 # (4)

Kafka Connect configuration (recommended)

config: # (5)
group.id: my-connect-cluster
offset.storage.topic: my-connect-cluster-offsets
config.storage.topic: my-connect-cluster-configs
status.storage.topic: my-connect-cluster-status
key.converter: org.apache.kafka.connect.json.JsonConverter
value.converter: org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable: true
value.converter.schemas.enable: true
config.storage.replication.factor: 3
offset.storage.replication.factor: 3
status.storage.replication.factor: 3

Authentication (optional)

authentication: # (7)
type: tls
certificateAndKey:
certificate: source.crt
key: source.key
secretName: my-user-source

TLS configuration (optional)

tls: # (8)
trustedCertificates:
- secretName: my-cluster-cluster-cert
pattern: "*.crt"

0 replies

charansaip · 2025-07-09T12:04:17Z

charansaip
Jul 9, 2025
Author

@im-konge can you please look now

9 replies

scholzj Jul 15, 2025
Maintainer

So, one thing to really double check is that:

You really enabled the TLS passthrough in Ingress as suggested by Kevin in the comment below. Please keep in mind that there is a flag in the Kubernetes Nginx Ingress to enable it in the first place. The annotation on the Ingress resource is not sufficient.
Assuming you double-checked / are sure about the previous point, you can double check it for example with openssl s_client -> when you connect to the broker URL with TLS-SNI, you should see that you get your broker certificate and not some Ingress certificate.
Assuming that seems to work as well, the next step is to look at the certificates you use. The kafk-custom-ca-secret in Connect has to contain the issuer of the kafka-custom-tls-secret certificate from the Kafka cluster. The user needs to be properly issued as well.

Some other things to keep in mind:

Is your Kafka cluster and your Connect cluster running on different Kube clusters? If not, you should not use an external ingress listener. It might not work within the same Kafka cluster and even if it does, it would be inefficient.
You should share the full logs from all the components. You say on the beginning that connector failed. If Connect works but a Connector fails, it might be a connector issue as well (or connector misconfiguration). Without the full logs it is also not clear what exactly is the problem. Server certificate? User certifictae? Hostname verification? Etc.

charansaip Jul 16, 2025
Author

@scholzj please check below.

And we are deploying kafka connect and kafka cluster in same k8 cluster and even ngninx alsoe deployed as pod in same cluster. And we enabled the tls pass thorough.

        args:
          - -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
          - -report-ingress-status
          - -default-server-tls-secret=kafka/kafka-custom-tls-secret 
          - -external-service=nginx-ingress
          - -enable-custom-resources=true
          - -enable-tls-passthrough
          - -ingress-class=nginx-kafka
          - -tls-passthrough-port=443

openssl s_client -connect kafka.appsuat.test.ie:443 -servername kafka.appsuat.test.ie

CONNECTED(00000003)
802B3A54737F0000:error:0A000458:SSL routines:ssl3_read_bytes:tlsv1 unrecognized name:../ssl/record/rec_layer_s3.c:1593:SSL alert number 112
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 333 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

2025-07-16 09:40:51,200 ERROR [AdminClient clientId=adminclient-1] Connection to node -1 (kafka.appsuat.test.ie/10.6.2.1:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2025-07-16 09:40:51,201 WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: unrecognized_name
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:365)
        at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:204)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
        at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736)
        at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)
        at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:518)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:373)
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:293)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:571)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1413)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1344)
        at java.base/java.lang.Thread.run(Thread.java:840)
2025-07-16 09:40:51,203 INFO App info kafka.admin.client for adminclient-1 unregistered (org.apache.kafka.common.utils.AppInfoParser) [kafka-admin-client-thread | adminclient-1]
2025-07-16 09:40:51,203 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: fetchMetadata
2025-07-16 09:40:51,203 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: fetchMetadata
2025-07-16 09:40:51,203 INFO [AdminClient clientId=adminclient-1] Timed out 2 remaining operation(s) during close. (org.apache.kafka.clients.admin.KafkaAdminClient) [kafka-admin-client-thread | adminclient-1]
2025-07-16 09:40:51,206 INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2025-07-16 09:40:51,206 INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2025-07-16 09:40:51,206 INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2025-07-16 09:40:51,206 ERROR Stopping due to error (org.apache.kafka.connect.cli.AbstractConnectCli) [main]
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
        at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:283)
        at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:263)
        at org.apache.kafka.connect.runtime.WorkerConfig.kafkaClusterId(WorkerConfig.java:393)
        at org.apache.kafka.connect.cli.AbstractConnectCli.startConnect(AbstractConnectCli.java:124)
        at org.apache.kafka.connect.cli.AbstractConnectCli.run(AbstractConnectCli.java:94)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:116)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
        at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
        at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
        at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
        at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:277)
        ... 5 more
Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: unrecognized_name

One more question since we are configuring custom certs. so basically Kafka should use custom certs not internal certs.
So we believe kafka still using internal even though with above brokercerts config. is there a way to config customer certs other than above i tried

scholzj Jul 16, 2025
Maintainer

3. One more question since we are configuring custom certs. so basically Kafka should use custom certs not internal certs.
So we believe kafka still using internal even though with above brokercerts config. is there a way to config customer certs other than above i tried

The openssl command should tell you that the correct certs are being used.

charansaip Jul 16, 2025
Author

@scholzj Could you able to found anything with above data.

scholzj Jul 16, 2025
Maintainer

Well, as I told you, you should check what certificate is used with the openssl command. You also did not explained how did you got the certificates for the various secrets - that is likely the issue.

kjbflood · 2025-07-14T17:31:45Z

kjbflood
Jul 14, 2025

Have you configured TLS passthru in the ingress?
https://arunsworld.medium.com/ssl-passthrough-via-kubernetes-ingress-b3eaf3c7c9da

1 reply

charansaip Jul 15, 2025
Author

yes i did

Strimzi

strimzi kafka operator using custom CA with nginx ingress controller #11627

Uh oh!

charansaip Jul 9, 2025

Replies: 5 comments · 10 replies

Uh oh!

im-konge Jul 9, 2025 Maintainer

Uh oh!

charansaip Jul 9, 2025 Author

Uh oh!

charansaip Jul 9, 2025 Author

Basic configuration (required)

Deployment specifications

Replicas (required)

Bootstrap servers (required)

Kafka Connect configuration (recommended)

Authentication (optional)

TLS configuration (optional)

Uh oh!

charansaip Jul 9, 2025 Author

Uh oh!

scholzj Jul 15, 2025 Maintainer

Uh oh!

charansaip Jul 16, 2025 Author

Uh oh!

scholzj Jul 16, 2025 Maintainer

Uh oh!

charansaip Jul 16, 2025 Author

Uh oh!

scholzj Jul 16, 2025 Maintainer

Uh oh!

kjbflood Jul 14, 2025

Uh oh!

charansaip Jul 15, 2025 Author

charansaip
Jul 9, 2025

Replies: 5 comments 10 replies

im-konge
Jul 9, 2025
Maintainer

charansaip
Jul 9, 2025
Author

charansaip
Jul 9, 2025
Author

charansaip
Jul 9, 2025
Author

scholzj Jul 15, 2025
Maintainer

charansaip Jul 16, 2025
Author

scholzj Jul 16, 2025
Maintainer

charansaip Jul 16, 2025
Author

scholzj Jul 16, 2025
Maintainer

kjbflood
Jul 14, 2025

charansaip Jul 15, 2025
Author