Worker segfaults in Docker after ~12s on DO

[Shy]

Hey Franz. I'm running into consistent segfaults when the worker runs in Docker/Linux environments (tested on DigitalOcean App Platform and locally) on a project.

Environment:

• swift-temporal-sdk: 0.1.0
• Swift: 6.2
• Base image: swift:6.2-noble (single-stage) or ubuntu:24.04 (multi-stage runtime)
• Platform: Linux ARM64 & x86_64

What's happening:

• Worker connects to Temporal Cloud (mTLS) successfully
• Server starts and reports ready
• Worker crashes with SIGSEGV ~12 seconds after "Starting Temporal worker..."
• Crash occurs in __swift_allocate_boxed_opaque_existential_1 during workflow execution

Crash logs (DigitalOcean):

[ INFO ] 👷 Starting Temporal worker... [attempt: 1]
swift-runtime: failed to suspend thread 1 while processing a crash; backtraces will be missing information
swift-runtime: failed to suspend thread 2 while processing a crash; backtraces will be missing information
[... many similar thread suspension failures ...]

*** Signal 11: Backtracing from 0x559a6d500e91... done ***

*** Program crashed: Bad pointer dereference at 0x0000100000010059 ***

Platform: x86_64 Linux (Ubuntu 24.04.3 LTS)

Thread 0 "tokio-runtime-w" crashed:

  0 [system]                0x0000559a6d500e91 __swift_allocate_boxed_opaque_existential_1 + 17
  1 [ra] [thunk]            0x0000559a6e3af801 thunk for @callee_guaranteed @async (@in_guaranteed Span) -> (@out A, @error @owned Error) + 128
  2 [ra]                    0x00007ea7f960ff50 swift::runJobInEstablishedExecutorContext(swift::Job*) + 367
  3 [ra]                    0x00007ea7f9610d47 swift_job_run_on_serial_and_task_executor + 86
  4 [ra] [inlined] [system] 0x0000559a6e45bd7f WorkflowTaskExecutor.run()
  5 [ra] [inlined]          0x0000559a6e45bd7f UnownedJob.runSynchronously(isolatedTo:taskExecutor:)
  6 [ra]                    0x0000559a6e45bd7f WorkflowInstance.runExecutor(context:) + 718 in WorkflowTaskExecutor.swift:74:21
  7 [ra]                    0x0000559a6e452af2 closure #2 in WorkflowInstance.run<A, B>(workflowType:activations:) + 81 in WorkflowInstance.swift:157:18
  8 [async]                 0x0000559a6e450be0 WorkflowInstance.run<A, B>(workflowType:activations:) in WorkflowInstance.swift:87

Attempted fixes:

1. Swift runtime flags (didn't fix DO crash):

ENV SWIFT_DETERMINISTIC_HASHING=1
ENV SWIFT_BACKTRACE=enable=no
ENV LIBDISPATCH_COOPERATIVE_POOL_STRICT=0

Helped locally but still crashes on DigitalOcean.

2. Multi-stage Docker build (different issue):
   Reduced image from 12.7GB → 2GB but introduces SSL certificate loading failures:

transportError: "Couldn't create SSL context" (cause: "failedToLoadCertificate")

Additional context:

• Single-stage Docker image is 12.7GB (includes full Swift toolchain)
• Running on 1GB RAM instance (DigitalOcean apps-s-1vcpu-1gb)
• Native macOS builds work fine
• Workflows are simple (single activity execution)

Curious if you have any work arounds for this one?

[Shy]

Ok so while, I've been working on getting API key authentication working with the Swift Temporal SDK I've been running in memory issues and I think it's all interconnected. Here's what I've found:

The Problem

When using API key auth, the worker crashes with a segfault (exit code 139) after running for 2-3 minutes. The crash happens specifically during worker startup/operation, not with direct client calls.

What I Investigated

Initial Approach - Passing API Key to Rust Core

I started by following the Ruby SDK's pattern and passing the API key to the Rust core via FFI:

Shy@8f07325

This immediately caused memory corruption with errors like:

freed pointer was not the last allocation

The app would crash instantly on macOS and hang indefinitely on Linux.

Second Attempt - Storing API Key in Variable First

Thought maybe it was a string lifetime issue, so I tried storing the API key in a local variable before passing it:

Shy@6766b0d

Same crash. Still memory corruption at the FFI boundary.

Third Attempt - Don't Pass to Rust, Use gRPC Metadata Instead

Since the Swift SDK uses grpc_override_callback, I tried:

1. Don't pass API key to Rust at all (use empty string): Shy@d2c7c50
2. Set the authorization: Bearer {api_key} header in Swift's gRPC metadata: Shy@4c9e5d1

This eliminated the "freed pointer" crash, but the worker STILL segfaults after 2-3 minutes in production.

Current Status

The app starts fine, HTTP server works, direct Temporal client calls work, but the worker crashes with exit code 139 (segfault) during operation. Logs show:

[ INFO ] 👷 Starting Temporal worker... [attempt: 1]
...2-3 minutes pass...
ERROR component jacob-party-server exited with code: 139

Comparison with mTLS

The mTLS had similar memory issues. I tried doing a rebuild of the swift-noble image as builder to drop the amount of resources being used. The image builds locally fine on my macbook, but on DO failed out when it came to compiling the swift ssl libraries (Shy/jacob.Party@b8b68b5). I also had a bunch of issues with getting the certificates actually into Digital ocean which lead me to moving away from it.

With API keys, workers crash every time after a few minutes.

My Theory

There's something broken with how the Swift SDK handles the FFI bridge when using API keys. Either:

1. The grpc_override_callback mechanism has memory safety issues
2. Some other FFI boundary issue I haven't figured out

The Ruby SDK passes API keys to Rust successfully, but Swift's implementation seems to have memory management bugs that mTLS doesn't trigger.

Have you seen anything like this before? Any ideas on what to check next?

[Shy]

I switched to testing on a bare DigitalOcean droplet (no Docker) because I got a fed up with docker and wanted to try something else. Using an ubuntu 24.04 build that I'm making via github actions because I was having build issues on my macbook for x86

I'm still getting the same results, server starts, worker connects to the cloud with mTLS using the official sdk (not my fork). Worker reports connected, accepts a workflow and then crashes with a SIGSEGV.

Same location every time:

Thread 2 crashed:
  0 [system] __swift_allocate_boxed_opaque_existential_1 + 17
  1 [ra] [thunk] thunk for @callee_guaranteed @async (@in_guaranteed Span) -> (@out A, @error @owned Error) + 128
  2 [ra] swift::runJobInEstablishedExecutorContext(swift::Job*) + 367 in libswift_Concurrency.so
  3 [ra] swift_job_run_on_serial_and_task_executor + 86
  4 [ra] [inlined] WorkflowTaskExecutor.run()
  5 [ra] [inlined] UnownedJob.runSynchronously(isolatedTo:taskExecutor:)
  6 [ra] WorkflowInstance.runExecutor(context:) + 718
      at .build/checkouts/swift-temporal-sdk/Sources/Temporal/Worker/Workflow/WorkflowTaskExecutor.swift:74:21
  7 [ra] closure #2 in WorkflowInstance.run<A, B>(workflowType:activations:) + 81
      at swift-temporal-sdk/Sources/Temporal/Worker/Workflow/WorkflowInstance.swift:157:18
  8 [async] WorkflowInstance.run<A, B>(workflowType:activations:)
      at swift-temporal-sdk/Sources/Temporal/Worker/Workflow/WorkflowInstance.swift:87

Everything works fine on my macbook. It's only an issue that I'm getting when I'm trying to run the server on Linux. Happy to provide more logs or test specific fixes if you have ideas on what to check next.

For now I think I'm calling it on this project and gonna move onto something else. If you wanna take a look at my code, it's the server directory on https://github.com/Shy/jacob.Party.

[Shy]

Also something to note is as I can tell workers use the Rust core, but direct client calls use a separate Swift implementation. I tried to resolve that in my fork of the sdk to add api key support, but I kept running into other issues so I abandoned it and reverted back to the main implementation y'all have put together.