The Operator sets the env var SSL_CERT_DIR is set to /tekton-custom-certs:/etc/ssl/certs:/etc/pki/tls/certs and mounts a certificate bundle at /tekton-custom-certs/ca-bundle.crt and /tekton-custom-certs/service-ca.crt. The OpenSSL docs state that SSL_CERT_DIR directories must "contain CA certificates in PEM format. The files each contain one CA certificate." The SSL Cert directories are not intended to contain certificate bundles, but individual certificates which were parsed from bundles using the openssl rehash command.
Strictly-compliant OpenSSL implementations do not work when SSL_CERT_DIR points to a directory containing only a certificate bundle, causing TLS to fail. Most software is not strictly compliant but some common software does break in this way, such as python's httpx library. As a result, even if no certificates are mounted at /tekton-custom-certs/, the httpx library can't use TLS on Fedora-based systems inside of a TaskRun.
We can fix the "TLS always fails" issue by setting SSL_CERT_DIR to the directory which openssl rehash populates on Fedora: /etc/pki/ca-trust/extracted/pem/directory-hash. However with this fix httpx will still not pickup any custom mounted certs. I'm not sure the best way to solve the issue and allow the custom certs to be picked up, and I'd like to get thoughts from the community on how to best address the issue.
Expected Behavior
The following task should execute and complete successfully:
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: use-httpx-for-tls
spec:
steps:
- name: fetch-google
image: <any fedora based image>
script: |
#!/bin/bash
# if necessary:
# $ dnf install python3 pip
# $ pip install httpx
python3 -c 'import httpx; httpx.get("https://google.com")'
Actual Behavior
The above taskrun exist with the following error:
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1147)
Additional Information
The source of the mixup appears to have originated from when the SSL_CERT_DIR paths were first added. The paths were copied from Golang's crypto/x509/root_linux package, however the crypto/x509 package is not OpenSSL compliant. See also: golang/go#79496
Steps to Reproduce the Problem
- Install Tekton with the Operator
- Run the command
python3 -c 'import httpx; httpx.get("https://google.com")' in any stepcontainer which uses a fedora/rhel based image
- observe the failure
- Run the command
SSL_CERT_DIR=/tekton-custom-certs:/etc/ssl/certs:/etc/pki/ca-trust/extracted/pem/directory-hash python3 -c 'import httpx; httpx.get("https://google.com")' in any stepcontainer which uses a fedora/rhel based image
- Observe the success
$ kubectl version
Client Version: v1.32.0
Kustomize Version: v5.5.0
Server Version: v1.30.13
$ tkn version
Client version: 0.44.1
Chains version: v0.26.3
Pipeline version: v1.9.3
Triggers version: v0.35.0
Dashboard version: v0.66.0
Operator version: v0.79.1
The Operator sets the env var
SSL_CERT_DIRis set to/tekton-custom-certs:/etc/ssl/certs:/etc/pki/tls/certsand mounts a certificate bundle at/tekton-custom-certs/ca-bundle.crtand/tekton-custom-certs/service-ca.crt. The OpenSSL docs state thatSSL_CERT_DIRdirectories must "contain CA certificates in PEM format. The files each contain one CA certificate." The SSL Cert directories are not intended to contain certificate bundles, but individual certificates which were parsed from bundles using theopenssl rehashcommand.Strictly-compliant OpenSSL implementations do not work when
SSL_CERT_DIRpoints to a directory containing only a certificate bundle, causing TLS to fail. Most software is not strictly compliant but some common software does break in this way, such as python'shttpxlibrary. As a result, even if no certificates are mounted at/tekton-custom-certs/, thehttpxlibrary can't use TLS on Fedora-based systems inside of a TaskRun.We can fix the "TLS always fails" issue by setting SSL_CERT_DIR to the directory which
openssl rehashpopulates on Fedora:/etc/pki/ca-trust/extracted/pem/directory-hash. However with this fixhttpxwill still not pickup any custom mounted certs. I'm not sure the best way to solve the issue and allow the custom certs to be picked up, and I'd like to get thoughts from the community on how to best address the issue.Expected Behavior
The following task should execute and complete successfully:
Actual Behavior
The above taskrun exist with the following error:
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1147)Additional Information
The source of the mixup appears to have originated from when the
SSL_CERT_DIRpaths were first added. The paths were copied from Golang's crypto/x509/root_linux package, however the crypto/x509 package is not OpenSSL compliant. See also: golang/go#79496Steps to Reproduce the Problem
python3 -c 'import httpx; httpx.get("https://google.com")'in any stepcontainer which uses a fedora/rhel based imageSSL_CERT_DIR=/tekton-custom-certs:/etc/ssl/certs:/etc/pki/ca-trust/extracted/pem/directory-hash python3 -c 'import httpx; httpx.get("https://google.com")'in any stepcontainer which uses a fedora/rhel based image