Skip to content

Injection of SSL_CERT_DIR is incorrect for Fedora-based images #3442

@aThorp96

Description

@aThorp96

The Operator sets the env var SSL_CERT_DIR is set to /tekton-custom-certs:/etc/ssl/certs:/etc/pki/tls/certs and mounts a certificate bundle at /tekton-custom-certs/ca-bundle.crt and /tekton-custom-certs/service-ca.crt. The OpenSSL docs state that SSL_CERT_DIR directories must "contain CA certificates in PEM format. The files each contain one CA certificate." The SSL Cert directories are not intended to contain certificate bundles, but individual certificates which were parsed from bundles using the openssl rehash command.

Strictly-compliant OpenSSL implementations do not work when SSL_CERT_DIR points to a directory containing only a certificate bundle, causing TLS to fail. Most software is not strictly compliant but some common software does break in this way, such as python's httpx library. As a result, even if no certificates are mounted at /tekton-custom-certs/, the httpx library can't use TLS on Fedora-based systems inside of a TaskRun.

We can fix the "TLS always fails" issue by setting SSL_CERT_DIR to the directory which openssl rehash populates on Fedora: /etc/pki/ca-trust/extracted/pem/directory-hash. However with this fix httpx will still not pickup any custom mounted certs. I'm not sure the best way to solve the issue and allow the custom certs to be picked up, and I'd like to get thoughts from the community on how to best address the issue.

Expected Behavior

The following task should execute and complete successfully:

apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: use-httpx-for-tls
spec:
  steps:
    - name: fetch-google
      image: <any fedora based image>
      script: |
        #!/bin/bash
        # if necessary:
        # $ dnf install python3 pip
        # $ pip install httpx
        python3 -c 'import httpx; httpx.get("https://google.com")'

Actual Behavior

The above taskrun exist with the following error:
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1147)

Additional Information

The source of the mixup appears to have originated from when the SSL_CERT_DIR paths were first added. The paths were copied from Golang's crypto/x509/root_linux package, however the crypto/x509 package is not OpenSSL compliant. See also: golang/go#79496

Steps to Reproduce the Problem

  1. Install Tekton with the Operator
  2. Run the command python3 -c 'import httpx; httpx.get("https://google.com")' in any stepcontainer which uses a fedora/rhel based image
  3. observe the failure
  4. Run the command SSL_CERT_DIR=/tekton-custom-certs:/etc/ssl/certs:/etc/pki/ca-trust/extracted/pem/directory-hash python3 -c 'import httpx; httpx.get("https://google.com")' in any stepcontainer which uses a fedora/rhel based image
  5. Observe the success
  • Kubernetes version:
$ kubectl version
Client Version: v1.32.0
Kustomize Version: v5.5.0
Server Version: v1.30.13
  • Tekton Pipeline version:
 $ tkn version
Client version: 0.44.1
Chains version: v0.26.3
Pipeline version: v1.9.3
Triggers version: v0.35.0
Dashboard version: v0.66.0
Operator version: v0.79.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions