TLS 1.2+1.3 server with an RSA key in Mbed TLS 4.0

[gilles-peskine-arm]

Consider a TLS server that has an RSA key and supports both TLS 1.2 and TLS 1.3. In TLS 1.2, RSA signatures use PKCS#1v1.5. In TLS 1.3, RSA signatures use PSS. Thus either the RSA private key must allow both PKCS#1v1.5 and PSS, or the server must have two separate key objects.

In Mbed TLS 3.6, the TLS server uses a single private key via the PK interface, and calls mbedtls_pk_sign_ext() to select PSS for TLS 1.3 (using the default PKCS#1v1.5 for TLS 1.2). This only works for a transparent PK key, not when the PK key is opaque (in the sense of MBEDTLS_PK_OPAQUE, i.e. backed by a PSA key). With an opaque key, the key's PSA policy determines which protocol is possible.

Mbed TLS 3.6 does allow a key to allow two signature algorithms, using the nonstandard PSA function psa_set_key_enrollment_algorithm(); we use that in ssl-opt.sh, but we don't document it, and it isn't something we want to promise in the long term because it's an Mbed TLS extension that other PSA implementations don't support.

In TF-PSA-Crypto 1.x, we want to change PK so that all RSA private keys would go through PSA, and thus the case-by-case algorithm selection done by mbedtls_pk_sign_ext() will no longer be possible. This means that a typical TLS server that parses an RSA key and calls mbedtls_ssl_conf_own_cert() will stop working.

The goal of this issue is to ensure that in TF-PSA-Crypto 1.0 and Mbed TLS 4.0:

1. There is a well-documented and tested way to set up a TLS 1.2+1.3 server with an RSA key.
2. If there's a reasonable way to set up a TLS 1.2+1.3 server with an RSA key that works in Mbed TLS 3.6, and we think it will stop working when PK in TF-PSA-Crypto 1.x moves further towards PSA, then this way must fail in Mbed TLS 4.0.