Add a `TimedCertificateReloader` #269

gjcairo · 2025-04-30T14:02:25Z

This PR adds a CertificateReloader protocol and a TimedCertificateReloader implementation.

A CertificateReloader allows for certificate-key pairs to be observed and updated. A TimedCertificateReloader does this every set interval of time.

This PR also provides an extension on TLSConfiguration to enable this reloading ability in NIO applications, so that updated certificates and keys can be used during SSL handshakes; as well as conformance of the TimedCertificateReloader to swift-service-lifecycle's Service protocol, for easy composition.

FranzBusch

This is exciting! Making TLS configuration updates easier is something that has been requested a lot

Sources/NIOCertificateHotReloading/NIOCertificateHotReloading.swift

FranzBusch · 2025-05-01T07:53:50Z

Sources/NIOCertificateHotReloading/NIOCertificateHotReloading.swift

+    /// A `NIOSSLContextConfigurationOverride` that will be used as part of the NIO application's TLS configuration.
+    /// Its certificate and private key will be kept up-to-date via whatever mechanism the specific ``CertificateReloader``
+    /// implementation provides.
+    var sslContextConfigurationOverride: NIOSSLContextConfigurationOverride { get async }


I have been thinking about this some more. There is two ways to go with this here. Keep it close to NIO then it should probably not be async but rather returning an ELF. Or this is a protocol that we see as a bridge to the Swift Concurrency ecosystem then we should make it async but I also think we should be returning a type here that is based on swift-certificates and map to NIOSSL types under the hood.

I think in general this should become a method though and not a computed property.

Would be interested to hear @Lukasa thoughts here

Isn't the intention for this to effectively be a current value? The reloader implementation does any work it actually needs be it async or otherwise.

In theory someone may want to avoid having a long running run for some reason?
Or another possible scenario would be if you were to yield new pairs into an async stream continuation from the run method, and iterate the stream in this getter instead of updating some [protected] state (like a LockedValueBox) containing the new values.

In any case, I do think that this is all avoidable and you can handle the asynchrony from elsewhere (like a run method). But not sure if Franz had something else in mind. I suppose having it async gives implementers some more freedom.

I'd prefer if it's not async. Otherwise calling it from synchronous code requires more complicated state management, etc.

I agree with George's assessment - my understanding was that this represents the "current" value. If so, I think it should be non-throwing, and sync.

It is meant to represent the current state yes. After this discussion and some more thought I'm also thinking I should revert this to be sync.

Sources/NIOCertificateHotReloading/NIOCertificateHotReloading.swift

glbrntt · 2025-05-01T10:18:59Z

Sources/NIOCertificateHotReloading/NIOCertificateHotReloading.swift

+    /// A `NIOSSLContextConfigurationOverride` that will be used as part of the NIO application's TLS configuration.
+    /// Its certificate and private key will be kept up-to-date via whatever mechanism the specific ``CertificateReloader``
+    /// implementation provides.
+    var sslContextConfigurationOverride: NIOSSLContextConfigurationOverride { get async }


Isn't the intention for this to effectively be a current value? The reloader implementation does any work it actually needs be it async or otherwise.

Sources/NIOCertificateHotReloading/NIOCertificateHotReloading.swift

glbrntt · 2025-05-01T12:15:33Z

Sources/NIOCertificateHotReloading/NIOCertificateHotReloading.swift

+    ///   - privateKeyDescription: A ``PrivateKeyDescription``.
+    @available(macOS 13, iOS 16, tvOS 16, watchOS 9, *)
+    public init(
+        refreshingEvery refreshInterval: Duration,


Do we run the risk of having ambiguity issues with the Duration/TimeAmount overload? Both have similar static funcs for e.g. .seconds etc.

I don't actually know the answer to this but at least on the tests (even when adding an availability annotation to match the Duration overload) it resolves to using TimeAmount. I am unsure if some other combination of factors may cause the compiler to become confused.

Sources/NIOCertificateHotReloading/NIOCertificateHotReloading.swift

hamzahrmalik · 2025-05-01T15:17:54Z

Sources/NIOCertificateHotReloading/NIOCertificateHotReloading.swift

+    /// A long-running method to run the ``TimedCertificateReloader`` and start observing updates for the certificate and
+    /// private key pair.
+    /// - Important: You *must* call this method to get certificate and key updates.
+    public func run() async throws {


this will throw CancellationError if cancelled during the sleep - is that intentional?

It is yeah - we want to make sure we respect cancellation. The interval may be set to a pretty high value (you likely only want to reload your certs every e.g. 10 minutes) so cancellation is important.

Sources/NIOCertificateHotReloading/NIOCertificateHotReloading.swift

Package.swift

czechboy0 · 2025-05-01T17:52:25Z

Sources/NIOCertificateHotReloading/NIOCertificateHotReloading.swift

+    /// private key pair.
+    /// - Important: You *must* call this method to get certificate and key updates.
+    public func run() async throws {
+        while !Task.isCancelled {


Might be better to do while true to keep behavior consistent regardless of when exactly cancelation happens. Otherwise sometimes you might end up with a clean return and with a thrown cancelation error otherwise.

Good point. Makes me wonder whether this should throw at all because with that change it can never return cleanly. (Not sure this actually matters much.)

Yeah it'd be good to get more guidance on the expected behavior - should services return or throw CancellationError. I've seen us do both.

Yeah good callout - I'll switch to while true.

With regards to whether it should be throwing or we should return, a (maybe useful?) data point, ServiceLifecycle will throw CancellationError if you try gracefully shutting down a service that's already been cancelled - even if the service itself didn't throw CancellationError when it was cancelled. Also, many of the async APIs Apple provides (like Task.sleep in this case, or async iterators' next) will handle cancellation by throwing a CancellationError too.

I think throwing out of this Task.sleep is particularly useful because you don't want to wait for the whole timeout period, which could be minutes, to finish before the reloading task is cancelled. I do think it's the right thing to do here.

czechboy0 · 2025-05-01T17:54:04Z

Sources/NIOCertificateHotReloading/NIOCertificateHotReloading.swift

+    ///   - refreshInterval: The interval at which attempts to update the certificate and private key should be made.
+    ///   - certificateDescription: A ``CertificateDescription``.
+    ///   - privateKeyDescription: A ``PrivateKeyDescription``.
+    public init(


Could we get an initializer that throws an error if it fails the first reload? Otherwise it might be harder to debug eg an incorrect path to the cert/key.

In other words - failing the first load is fatal, and the server probably wants to fail to start. Failing to reload later can be handled more gracefully - just keep the old value.

It'd be great to make that distinction easier to react to in a server startup sequence. One way is to represent this as a throwing initializer that performs (or fails) the first load.

Could we get an initializer that throws an error if it fails the first reload? Otherwise it might be harder to debug eg an incorrect path to the cert/key.

I originally flip flopped a bit on this. I ended up landing on not failing because what if your certificate is not yet present but will be? You could have a daemon that takes care of managing your TLS certificates and this could be the first time you're starting your service on a machine and the cert is not yet present, or the cert could be missing for whatever other reason (data corruption, mistakenly deleted, etc).

I was trying to find what other projects do in this scenario in case we want to get some inspiration. cert-manager (link) allows for this possibility and does not fail on first load.

However it isn't lost on me that not failing could potentially make debugging harder. It could also mean an insecure connection might be established with a client which obviously isn't good. So I'm okay with making this init throwing. I just want to make sure we've considered both options.

Yeah good on you for being thoughtful on this, and I'm not even saying there shouldn't be a graceful initializer that ignores the first failure as well. I mainly just wanted to have the option to have a throwing initializer.

I think it's reasonable to offer this, I'll add additional inits that validate they can be loaded.

im somewhat suspicious that there's a way to break this if you write

init(refreshInterval: .seconds(1)...
the compiler won't know if you meant Duration.seconds or TimeAmount.seconds

That said, i can't come up with a way to make this break

im thinking about apple/swift-metrics#144

Because I had to up the availability requirements to use the async timer sequence, we can just leave the Duration init - I got rid of the TimeAmount overload.

czechboy0 · 2025-05-01T17:55:39Z

Sources/NIOCertificateHotReloading/NIOCertificateHotReloading.swift

+    /// A `NIOSSLContextConfigurationOverride` that will be used as part of the NIO application's TLS configuration.
+    /// Its certificate and private key will be kept up-to-date via whatever mechanism the specific ``CertificateReloader``
+    /// implementation provides.
+    var sslContextConfigurationOverride: NIOSSLContextConfigurationOverride { get async }


I'd prefer if it's not async. Otherwise calling it from synchronous code requires more complicated state management, etc.

czechboy0 · 2025-05-01T20:05:33Z

Sources/NIOCertificateHotReloading/NIOCertificateHotReloading.swift

+        let certificate: Certificate?
+        switch self.certificateDescription.format._backing {
+        case .der:
+            certificate = try? Certificate(derEncoded: certificateBytes)


We should let the error be thrown out and at least logged (here and all the try? below as well)

I am not sure about throwing - I don't think we want the reloader to stop running when some potentially transient error is encountered.

We could have a max number of failed attempts though, and throw after that number has been reached.

W.r.t logging, we don't currently depend on swift-log in this package, so I'm not sure if that's something we want. I guess we could also just print (which I know is not ideal but it's what we already do for the debug events handler)

Okay so given the other discussion about having throwing inits, I have made these methods throw, but I am just ignoring the errors from the main reload loop.

Still open questions about having a max number of failed attempts and a logger.

hamzahrmalik · 2025-05-02T17:20:54Z

What's the best way to use this with a server?

let tlsConfig = TLSConfiguration.makeServerConfiguration(certificateChain: ??????) // What do i pass here?
tlsConfig.sslContextCallback = { _, promise in
    promise.succeed(timedCertificateReloader.sslContextConfigurationOverride)
}

It'd be nice if TimedCertificateReloader could expose the initial config, since it already read it in its initializer, or if TLSConfiguration could be made without an initial cert-key pair (and rely on the override).

Or do i have to just manually read the first cert-key pair like:

guard let certificateChain = certificateReloader.sslContextConfigurationOverride.certificateChain,
              let privateKey = certificateReloader.sslContextConfigurationOverride.privateKey else {
            throw ConfigurationError.invalidValue("Certificate or key path is invalid")
        }
        var tlsConfiguration = TLSConfiguration.makeServerConfiguration(certificateChain: certificateChain, privateKey: privateKey)
        tlsConfiguration.sslContextCallback = { _, promise in
            // This callback is called for every TLS hello and we'll use the latest cert from the provider.
            promise.succeed(certificateReloader.sslContextConfigurationOverride)
        }

gjcairo · 2025-05-06T15:47:53Z

@hamzahrmalik thanks for bringing this up - I've improved the docs/added some examples, and also added a new makeServerConfiguration(certificateReloader:) extension to make it easier for server side configurations.

hamzahrmalik · 2025-05-06T18:27:26Z

Sources/NIOCertificateReloading/CertificateReloader.swift

+    /// Configure a ``CertificateReloader`` to observe updates for the certificate and key pair used.
+    /// - Parameter reloader: A ``CertificateReloader`` to watch for certificate and key pair updates.
+    /// - Returns: A ``NIOSSL/TLSConfiguration`` that reloads the certificate and key used in its SSL handshake.
+    mutating public func setCertificateReloader(_ reloader: some CertificateReloader) -> Self {


this is mildly confusing to me, it's mutating but also returns self

If the return is just for convenience, then we should maybe make it discardable result

Or was it intended to not mutate the original and make a copy?

also, xcode says you need to import NIOCore

Instance method 'succeed' is not available due to missing import of defining module 'NIOCore'

It's for convenience, so you can chain calls. I've added @discardableResult.

I'm with @hamzahrmalik on this one: it's a bit confusing.

Chaining calls isn't a common pattern for config structs or for TLSConfiguration so this shouldn't return Self.

czechboy0 · 2025-05-06T19:12:58Z

Sources/NIOCertificateReloading/TimedCertificateReloader.swift

+    public func run() async throws {
+        for try await _ in AsyncTimerSequence.repeating(every: self.refreshInterval).cancelOnGracefulShutdown() {
+            // We don't want to throw out of this method so simply ignore errors thrown from `reloadPair`.
+            try? self.reloadPair()


Could benefit from some logging integration, at least a callback when reload fails? I'd want to hook it into logging/metrics when running in a service.

Added logging when the reloading fails.

bla

czechboy0

Thanks, Gus!

Sources/NIOCertificateReloading/TimedCertificateReloader.swift

Package.swift

glbrntt · 2025-05-07T10:19:21Z

Sources/NIOCertificateReloading/CertificateReloader.swift

+    /// Configure a ``CertificateReloader`` to observe updates for the certificate and key pair used.
+    /// - Parameter reloader: A ``CertificateReloader`` to watch for certificate and key pair updates.
+    /// - Returns: A ``NIOSSL/TLSConfiguration`` that reloads the certificate and key used in its SSL handshake.
+    mutating public func setCertificateReloader(_ reloader: some CertificateReloader) -> Self {


I'm with @hamzahrmalik on this one: it's a bit confusing.

Chaining calls isn't a common pattern for config structs or for TLSConfiguration so this shouldn't return Self.

Sources/NIOCertificateReloading/CertificateReloader.swift

Sources/NIOCertificateReloading/TimedCertificateReloader.swift

glbrntt · 2025-05-07T12:15:11Z

Sources/NIOCertificateReloading/TimedCertificateReloader.swift

+        )
+    }
+
+    /// Attempt to initialize a new ``TimedCertificateReloader``, but throw if the given certificate and private keys cannot be


Can I propose that rather than having inits which also do the first refresh and can throw that we just have the non-throwing inits and also expose a refreshNow() throwing func?

This means that:

the init doesn't have side effects

refreshing is explicit

any throwing is directly tied to the refresh operation (throwing could come from something else in the init but as a user I don't know that unless I read the code)

we can remove some duplication

Generally speaking, if work should be done after creating the object it should either be a separate func which can be made more convenient by a static func which creates and then runs that func before returning the object.

So are you suggesting that on top of the reloadNow() function, we also provide a static TimedCertificateReloader.makeReloaderAndValidateLocations(...) (naming TBC)?

@czechboy0 - what do you think of this change? Tagging you since you originally asked for throwing inits.

Personally, if we also provide a static method that does both things, I'm okay with it.

I (personally) wouldn't bother adding the static method as well but I have no objections to it being added.

I'm fine with either a throwing init or a throwing static func (they're mostly equivalent in my mind), I just want it to be a single call that creates and returns the instance, and we know the instance was able to load the certs at least once (meaning we don't need code that handles one of these that never successfully loaded certs).

I suspect we'll add that call to many of our examples and docs, as I believe that a service that couldn't load certs at least once shouldn't ever mark itself healthy. But I'm okay with that not being the only usage pattern here, in case there are use cases where someone wants to keep running even without certs ever loading successfully.

Not being able to load the certs on the first try doesn't mean they won't ever load successfully :) but I agree this could be a common pattern so I do think we should offer a way to do so easily.

Because of the switch from while true in the main loop to using the timer sequence, I've had to increase the platforms in the availability guards anyways. So I'll just leave the non-throwing init using Duration (and get rid of the TimeAmount overload, since it was only there to support older platforms which can't be supported anymore), add a new reloadNow() func, and add a new static method to init and reload.

Could you illustrate with a code snippet how this composes in a real-world example? When I tried thinking this through, it becomes very convoluted and hard to understand when exactly the server is healthy and running.

Can we at least go back to the static function that does the creation and running the first fetch? We'll just document that it's the recommended one in practice, that's fine.

Just making up APIs here, but hopefully it's clear enough:

let reloader = CertificateReloader(refreshInterval: .minutes(60)) let server = SomeServer(tls: .makeServerConfig(using: reloader)) let serviceGroup = ServiceGroup(services: [reloader, server]) try await serviceGroup.run()

If reloader throws on run() (which should only be possible when doing the first refresh) then the service group will get shutdown almost immediately. I think there's a small timing window between the reloader starting and failing where the server could start but would be a lame duck. In practice this is unlikely to be an issue as the server should be failing its health checks and the timing window before shutting down should be very small.

Is this meant to be the same makeServerConfiguration as in this PR? Because that's a throwing function that will throw an error if the cert hasn't been loaded yet, so it can only be practically used with a CertificateReloader that is guaranteed to have already loaded certs (my preference).

Ah right, yeah I misremembered the API. Let's just go with the static, it's easy to understand.

Cool, if we also get the makeClient variant, I'm happy with this PR.

Sources/NIOCertificateReloading/TimedCertificateReloader.swift

glbrntt · 2025-05-07T12:19:59Z

Sources/NIOCertificateReloading/TimedCertificateReloader.swift

+                    private key pair.
+                    """,
+                    metadata: [
+                        "error": error


Can you add tests for the description of the errors that can be thrown? Structs wrapping enums have bad default descriptions IIRC.

Tests/NIOCertificateReloadingTests/TimedCertificateReloaderTests.swift

…esent

Sources/NIOCertificateReloading/CertificateReloader.swift

czechboy0

Looks great, thank you @gjcairo 🙏

glbrntt · 2025-05-13T09:46:21Z

Sources/NIOCertificateReloading/CertificateReloader.swift

+    ) throws -> Self {
+        let override = certificateReloader.sslContextConfigurationOverride
+
+        guard override.certificateChain != nil else {


Should we be setting the cert chain and private key on the config below?

Hm, I'm actually not sure it's needed since the reloader takes care of setting the override via sslContextCallback, but I'll set it just in case.

Sources/NIOCertificateReloading/CertificateReloader.swift

glbrntt · 2025-05-13T09:49:25Z

Sources/NIOCertificateReloading/TimedCertificateReloader.swift

+#if compiler(>=6.0)
+@available(macOS 13, iOS 16, watchOS 9, tvOS 16, macCatalyst 16, visionOS 1, *)
+#else


Potentially silly question: is this actually required? Doesn't the * in the availability in the #else branch cover visionOS?

Good point - I think that should be fine yes. Do we care about explicitly spelling out all platforms or not really? I know we've been adding visionOS to other packages, because we weren't using *, but should we opted to use * instead? Or do we favour listing them explicitly?

I don't think so. I'm not opposed to spelling it out fully, it just seems overkill to compiler guard it (especially if it was applied more widely)

Sources/NIOCertificateReloading/TimedCertificateReloader.swift

glbrntt

docs nit but LGTM otherwise

Sources/NIOCertificateReloading/TimedCertificateReloader.swift

FranzBusch reviewed Apr 30, 2025

View reviewed changes

gjcairo requested a review from FranzBusch April 30, 2025 16:18

FranzBusch reviewed May 1, 2025

View reviewed changes

glbrntt reviewed May 1, 2025

View reviewed changes

gjcairo added the 🆕 semver/minor Adds new public API. label May 1, 2025

hamzahrmalik reviewed May 1, 2025

View reviewed changes

czechboy0 reviewed May 1, 2025

View reviewed changes

gjcairo force-pushed the cert-reloader branch from 7e88f84 to 000d04d Compare May 6, 2025 15:44

hamzahrmalik reviewed May 6, 2025

View reviewed changes

czechboy0 reviewed May 6, 2025

View reviewed changes

gjcairo added 16 commits May 7, 2025 10:33

Add a TimedCertificateReloader

554f22e

Remove unneeded imports

6b6a5b5

Refactor some stuff

7a755d7

Add conformance to ServiceLifecycle/Service

744d803

Format

0dae5b2

Fix broken imports

617d1ac

Add more docs

222a050

Make sslContextConfigurationOverride async

09520a7

Fix imports

b8eb9fb

Add init using Duration

ca8e42a

PR comments

65a4600

Format

c9402e9

Add missing import

76a3dc5

Fix docs

452b193

Add missing import

ef88924

Rename targets and move around some code

ca5962e

bla

gjcairo added 5 commits May 7, 2025 10:33

Add docs and a new TLSConfig init

ed45e53

Formatting

9771210

Fix availability guards

6eb1e0c

Annotate setCertificateReloader with @discardableResult

47d25bb

Add missing import

9e3d133

gjcairo force-pushed the cert-reloader branch from 8798c3b to 9e3d133 Compare May 7, 2025 09:33

gjcairo added 2 commits May 7, 2025 10:58

Format

eb78ed3

Add logging when reloading fails

8fbba2d

czechboy0 reviewed May 7, 2025

View reviewed changes

Sources/NIOCertificateReloading/TimedCertificateReloader.swift Outdated Show resolved Hide resolved

Sources/NIOCertificateReloading/TimedCertificateReloader.swift Outdated Show resolved Hide resolved

Sources/NIOCertificateReloading/TimedCertificateReloader.swift Show resolved Hide resolved

glbrntt reviewed May 7, 2025

View reviewed changes

gjcairo added 7 commits May 7, 2025 14:32

PR changes

2f52c35

Rename CertificateDescription to CertificateSource

fa65c5b

Rename PrivateKeyDescription to PrivateKeySource

516c61c

Rename some missing CertificateDescription arguments

f3cce7f

PR changes

33fe740

Improve docs

d8c0f43

Return NIOSSLContextConfigurationOverride.noChanges when no pair pr…

87b0335

…esent

czechboy0 reviewed May 9, 2025

View reviewed changes

Sources/NIOCertificateReloading/CertificateReloader.swift Show resolved Hide resolved

PR changes

c34a816

czechboy0 approved these changes May 12, 2025

View reviewed changes

gjcairo requested a review from glbrntt May 12, 2025 16:43

glbrntt reviewed May 13, 2025

View reviewed changes

Set cert and pkey in client config

6570045

gjcairo requested a review from glbrntt May 13, 2025 15:26

glbrntt approved these changes May 13, 2025

View reviewed changes

Sources/NIOCertificateReloading/TimedCertificateReloader.swift Outdated Show resolved Hide resolved

gjcairo force-pushed the cert-reloader branch from bf7d104 to 781396e Compare May 14, 2025 09:30

PR changes

2852c84

gjcairo force-pushed the cert-reloader branch from 781396e to 2852c84 Compare May 14, 2025 09:32

gjcairo merged commit 949cf2d into apple:main May 14, 2025
25 of 26 checks passed

gjcairo deleted the cert-reloader branch May 14, 2025 12:37

Add a TimedCertificateReloader #269

Add a TimedCertificateReloader #269

Uh oh!

Conversation

gjcairo commented Apr 30, 2025

Uh oh!

FranzBusch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

czechboy0 May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hamzahrmalik commented May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Add a `TimedCertificateReloader` #269

Add a `TimedCertificateReloader` #269

czechboy0 May 1, 2025 •

edited

Loading

hamzahrmalik commented May 2, 2025 •

edited

Loading