Skip to content

fix: Resolve libcurl symbols via dlsym so Scala Native binaries without CurlBindings link (#622) #636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
xerial merged 1 commit into from
Jul 6, 2026
Merged

fix: Resolve libcurl symbols via dlsym so Scala Native binaries without CurlBindings link (#622) #636

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 123 additions & 0 deletions adr/2026-07-06-curl-shim-weak-linking.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# 2026-07-06: `uni_curl_shim.c` resolves libcurl symbols via `dlsym`, not `extern`

Issue: https://github.com/wvlet/uni/issues/622

## Context

Scala Native cannot reliably call libcurl's variadic `curl_easy_setopt` /
`curl_easy_getinfo` — see the file header in `uni_curl_shim.c` and the
`CurlBindings.Extern` docstring. PR #580 (v2026.1.13) fixed that by routing
through fixed-arity C wrappers compiled from
`uni/.native/src/main/resources/scala-native/uni_curl_shim.c`, letting the C
compiler emit the correct variadic calling convention at the shim's own call
site.

The regression #622 reports: any Scala Native project pulling in
`uni_native0.5_3-2026.1.13` and later — even one whose Scala code only uses
`LogSupport` — fails to link with

ld: uni_curl_shim.c.o: undefined reference to `curl_easy_setopt'
ld: uni_curl_shim.c.o: undefined reference to `curl_easy_getinfo'

Why: Scala Native's linker compiles **every** `.c` file under
`resources/scala-native/` in **every** jar on the classpath, unconditionally.
The v2026.1.13 shim declared `extern int curl_easy_setopt(...)`, so its `.o`
carried a strong undefined reference. `-lcurl` reaches the linker only through
`@link("curl")` on `CurlBindings.Extern`; when Scala DCE removes `CurlBindings`
(because nothing references it), that link option never propagates, and the
undefined reference in the always-compiled `.o` blows up the final link. The
`-lcurl` in this repo's own `build.sbt` only helps this build — downstream
projects don't see it. The shim can't sit in a "curl users only" sub-module
either: Scala Native has no jar-level knob to conditionally include a resource
`.c`.

## Decision

Resolve the libcurl symbols lazily via
`dlsym(RTLD_DEFAULT, "curl_easy_setopt")` — not via C-level `extern` — inside a
`pthread_once` initialiser that stores each pointer in a `static` cache. The
shim's `.o` then has zero references to libcurl symbols; downstream projects
that don't use CurlBindings link cleanly. Projects that do use CurlBindings
still pull libcurl in via `@link("curl")` on `CurlBindings.Extern`, and
`RTLD_DEFAULT` finds the symbols inside the already-loaded libcurl at runtime.

The function-pointer typedefs are declared variadic
(`typedef int (*fn)(void *, int, ...)`) so the call at the shim's own call site
still emits the correct variadic calling convention — that's what fixes the
original CURLE_URL_MALFORMAT bug from #580, and it works exactly the same via a
variadic function-pointer as via a variadic extern.

If `dlsym` returns NULL (the shim is called from a build that somehow got the
wrappers linked in without libcurl present), the shim prints a pointer to #622
on stderr and `abort()`s rather than jumping to NULL.

## Non-obvious points a future reader would otherwise reverse-engineer

### `weak_import` / `__attribute__((weak))` on the extern is *not* a substitute

The obvious alternative — mark the `extern` declarations weak so the linker
tolerates the undefined — appears to work on Linux but **fails on macOS**. macOS
`ld` still refuses to leave a weak-undefined symbol unresolved at final link
time unless the binary is linked with `-undefined dynamic_lookup` (or the
per-symbol `-U <sym>`). That flag can't be forced on downstream binaries from a
jar-resource `.c` file, so the trick is unusable here. `dlsym` sidesteps the
whole problem because the `.o` carries no libcurl symbol at all — only libc
symbols (`dlsym`, `fprintf`, `abort`, `___stderrp`), which are guaranteed
available.

### Why `RTLD_DEFAULT` — not `dlopen("libcurl", ...)`

`RTLD_DEFAULT` walks the symbols already loaded into the process. That's the
right thing here: consumers that use CurlBindings get libcurl loaded via
`@link("curl")`, and any curl functions Scala calls directly (e.g.
`Extern.easyInit`) must resolve to the *same* libcurl that the shim uses —
otherwise CURL handles would be exchanged between two different libcurl
instances. `dlopen("libcurl.so.4", ...)` would risk loading a second copy.

### The `-ldl` / `-lpthread` question

`dlsym` lives in `<dlfcn.h>`, `pthread_once` in `<pthread.h>`. On macOS both are
in libSystem (always linked). On modern Linux (glibc 2.34+, released 2021)
`libdl` and `libpthread` are merged into `libc`; on older glibc they were
separate — but Scala Native's own `nativelib` already links both, so the shim
inherits them. No extra linker option is required from consumers. Verified by
inspecting Scala Native's link line (`[pthread, dl, m, crypto, curl, z]`).

### `_GNU_SOURCE` is required on glibc

`RTLD_DEFAULT` is a GNU extension: glibc's `<dlfcn.h>` only defines it when
`_GNU_SOURCE` is set. macOS libSystem exposes it unconditionally, so the macro
is harmless there. The file defines `_GNU_SOURCE` at the top — dropping it
would silently break Linux builds with `error: 'RTLD_DEFAULT' undeclared`.

### Lazy init needs `pthread_once`, not "same value written twice"

An earlier draft argued no lock was needed because racing threads would race to
write the same pointer value. That reasoning is wrong under C11: concurrent
unsynchronised accesses to a non-atomic object where at least one is a write is
a data race, i.e. undefined behaviour, regardless of the value written. The
compiler is entitled to assume no such race exists and to re-order/eliminate
the loads and stores accordingly. `pthread_once` gives a proper
happens-before edge between the resolver's writes and every subsequent reader,
at negligible cost after the first call (a plain "already done" flag check).

### The variadic-typedef trick is the whole variadic story

The C ABI's variadic calling convention is determined by the *type at the call
site*, not by whatever the symbol on the other end was compiled as. That is why
calling through `uni_curl_setopt_fn` (declared variadic) reproduces #580's fix
even though the pointer was obtained from `dlsym` (which has no type
information). Do not "simplify" the typedef to a fixed-arity signature — that
reintroduces the CURLE_URL_MALFORMAT bug from #580.

## Consequences

- Downstream Scala Native projects that don't use CurlBindings link cleanly
again — #622 fixed.
- Downstream projects that do use CurlBindings pay one `dlsym` per unique
wrapper on first call (three lookups total across the process's lifetime);
every subsequent call is a plain indirect function call.
- CurlBindings' `@link("curl")` remains load-bearing — it's how libcurl actually
gets into the process. Do not remove it.
- Removing the shim `.c` file, or any `dlfcn`/`stdio` include, would regress
either the ABI fix or the link-error fix; keep both.
1 change: 1 addition & 0 deletions adr/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,4 @@ Read the relevant ADR before modifying the area it covers. Add a new entry here
- [`2026-06-30-sbt-uni-crossproject.md`](2026-06-30-sbt-uni-crossproject.md) — the `sbt-uni-crossproject/` build: a minimal, uni-owned sbt 2.x re-implementation of `portable-scala/sbt-crossproject` (which isn't ported to sbt 2.x), supporting only the `CrossType.Pure` layout uni uses. Read before touching that build; covers the single-plugin-for-all-three-platforms choice, the Scala 3 val-name macro replicating sbt's `KeyMacro.definingValName`, why internal materialization needs `new CrossProject(...)`, and the `given Conversion[Builder, CrossProject]` build trigger.
- [`2026-06-30-sbt2-main-build-migration.md`](2026-06-30-sbt2-main-build-migration.md) — migrating the **main build** to sbt 2.x: swaps the unported third-party plugins for the uni-owned ones (`sbt-uni-crossproject`, `uni-jsenv-playwright`, `sbt-uni` for `sbt-revolver`). Read before touching `build.sbt` / `project/plugin.sbt`; covers the output-dir name collision (root → `uni-root`), `%%%`→`%%` and the `scalajs-test-interface_2.13` single-`%` exception, `Def.uncached` for `jsEnv`, and the `implicitConversions` import.
- [`2026-07-06-plugin-extension-points.md`](2026-07-06-plugin-extension-points.md) — `wvlet.uni.plugin` is built on typed `ExtensionPoint`s (identity-compared singletons; keyed points reject duplicate ids at activation), with `PluginContext` reduced to `contribute` + `onDeactivate`. Read before adding new contribution kinds: define a point next to the contributed type (`Command.point`, `RPCPlugin.routerPoint`) so dependency arrows point into `plugin`, never out of it.
- [`2026-07-06-curl-shim-weak-linking.md`](2026-07-06-curl-shim-weak-linking.md) — `uni_curl_shim.c` resolves libcurl's variadic setopt/getinfo via `dlsym(RTLD_DEFAULT, ...)` inside a `pthread_once` init, not C-level `extern`s, so downstream Scala Native builds that don't use `CurlBindings` still link (issue #622). Read before touching the shim: covers why `__attribute__((weak))` on the extern fails on macOS, why `RTLD_DEFAULT` (not `dlopen`) is the right lookup, why the variadic-typedef ABI trick from #580 is still load-bearing, why `_GNU_SOURCE` is required on glibc, and why lazy init needs `pthread_once` not just "same value written twice".
70 changes: 62 additions & 8 deletions uni/.native/src/main/resources/scala-native/uni_curl_shim.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,24 +20,78 @@
* arm64-apple-darwin), and a CVarArg* extern does not pass the argument through at all on this
* toolchain. Both make curl read garbage for the value (e.g. a valid URL reported as
* CURLE_URL_MALFORMAT). Routing through these real, fixed-arity C functions lets the C compiler emit
* the correct variadic call, while Scala Native sees ordinary fixed-arity symbols.
* the correct variadic call, while Scala Native sees ordinary fixed-arity symbols. What matters is
* the *declared* type at the call site — variadic function-pointer typedefs give us the right ABI
* regardless of how the pointer was obtained.
*
* The curl prototypes are declared locally so the shim needs no libcurl headers at build time; the
* symbols are resolved from the linked libcurl (-lcurl). CURLoption/CURLINFO are C enums (int).
* The libcurl symbols are resolved lazily via dlsym(RTLD_DEFAULT, ...) inside a pthread_once init,
* rather than declared as C-level `extern`s. Scala Native compiles this file into every downstream
* project that has the uni-native jar on its classpath, whether or not the project references
* CurlBindings. With a strong `extern int curl_easy_setopt(...)` the compiled .o carries an
* unresolved reference to `curl_easy_setopt`; downstream projects that don't reference CurlBindings
* never get -lcurl (its @link("curl") is dead-code-eliminated) and their link fails with "undefined
* reference to curl_easy_setopt" (issue #622). Using dlsym leaves this .o free of libcurl symbol
* references, so downstream links succeed. When CurlBindings *is* used, its @link("curl") pulls
* libcurl into the binary at load time, so dlsym(RTLD_DEFAULT, "curl_easy_setopt") finds it —
* RTLD_DEFAULT walks all libraries loaded into the process. libdl and libpthread are already linked
* by Scala Native's nativelib, so no extra linker flag is needed.
*/

extern int curl_easy_setopt(void *handle, int option, ...);
extern int curl_easy_getinfo(void *handle, int info, ...);
/* On glibc, RTLD_DEFAULT is exposed by <dlfcn.h> only under _GNU_SOURCE. macOS libSystem exposes it
* unconditionally; defining the macro there is harmless. */
#define _GNU_SOURCE

#include <dlfcn.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
Comment on lines +44 to +47

@gemini-code-assist gemini-code-assist Bot Jul 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

On Linux systems using glibc, the RTLD_DEFAULT constant is a GNU extension and is only exposed in <dlfcn.h> if the _GNU_SOURCE feature test macro is defined before including the header. To ensure portable compilation across all Linux distributions and toolchains, define _GNU_SOURCE at the very top of the file.

#define _GNU_SOURCE
#include <dlfcn.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>

@xerial xerial Jul 6, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed — #define _GNU_SOURCE is now the first line of the file, before <dlfcn.h>. On macOS libSystem defines RTLD_DEFAULT unconditionally so the macro is harmless there.


/*
* Variadic function-pointer typedefs. Calling through these yields the correct variadic ABI at the
* call site (arg on stack on arm64-apple-darwin), which is the whole reason this shim exists.
*/
typedef int (*uni_curl_setopt_fn)(void *handle, int option, ...);
typedef int (*uni_curl_getinfo_fn)(void *handle, int info, ...);

static uni_curl_setopt_fn uni_curl_setopt_p = NULL;
static uni_curl_getinfo_fn uni_curl_getinfo_p = NULL;

/* pthread_once guarantees the resolver runs exactly once even under concurrent first calls; without
* it, racing writes to the static function pointers are a data race under C11 (undefined behaviour)
* regardless of whether the racing threads write the same value. */
static pthread_once_t uni_curl_shim_init_once = PTHREAD_ONCE_INIT;

static void *uni_curl_shim_resolve(const char *name) {
void *sym = dlsym(RTLD_DEFAULT, name);
if (sym == NULL) {
fprintf(
stderr,
"uni_curl_shim: libcurl symbol '%s' not found in this process. Add "
"wvlet.uni.http.CurlBindings to your Scala Native build (which links "
"libcurl via @link(\"curl\")), or install libcurl. See wvlet/uni#622.\n",
name);
abort();
}
return sym;
}

static void uni_curl_shim_init(void) {
uni_curl_setopt_p = (uni_curl_setopt_fn)uni_curl_shim_resolve("curl_easy_setopt");
uni_curl_getinfo_p = (uni_curl_getinfo_fn)uni_curl_shim_resolve("curl_easy_getinfo");
}

int uni_curl_easy_setopt_ptr(void *handle, int option, void *value) {
return curl_easy_setopt(handle, option, value);
pthread_once(&uni_curl_shim_init_once, uni_curl_shim_init);
return uni_curl_setopt_p(handle, option, value);
}

int uni_curl_easy_setopt_long(void *handle, int option, long value) {
return curl_easy_setopt(handle, option, value);
pthread_once(&uni_curl_shim_init_once, uni_curl_shim_init);
return uni_curl_setopt_p(handle, option, value);
}

/* Typed long* out-parameter for CURLINFO_*_RESPONSE_CODE and other `long` infos. */
int uni_curl_easy_getinfo_long(void *handle, int info, long *value) {
return curl_easy_getinfo(handle, info, value);
pthread_once(&uni_curl_shim_init_once, uni_curl_shim_init);
return uni_curl_getinfo_p(handle, info, value);
}
Comment on lines +56 to 97

@gemini-code-assist gemini-code-assist Bot Jul 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The current lazy initialization of uni_curl_setopt_p and uni_curl_getinfo_p is not thread-safe. If multiple threads concurrently invoke the shim functions when the pointers are NULL, they will concurrently read and write to these static variables. In C11, concurrent unsynchronized accesses to non-atomic variables where at least one is a write constitutes a data race, which is undefined behavior (UB). Even if they write the same value, compiler optimizations can lead to unexpected behavior.

To resolve this portably and efficiently without adding complex locks or relying on compiler-specific atomics, use POSIX's pthread_once. This guarantees thread-safe, one-time initialization of both function pointers.

static uni_curl_setopt_fn  uni_curl_setopt_p  = NULL;
static uni_curl_getinfo_fn uni_curl_getinfo_p = NULL;
static pthread_once_t uni_curl_init_once = PTHREAD_ONCE_INIT;

static void *uni_curl_shim_resolve(const char *name) {
  void *sym = dlsym(RTLD_DEFAULT, name);
  if (sym == NULL) {
    fprintf(
        stderr,
        "uni_curl_shim: libcurl symbol '%s' not found in this process. Add "
        "wvlet.uni.http.CurlBindings to your Scala Native build (which links "
        "libcurl via @link(\\\"curl\\\")), or install libcurl. See wvlet/uni#622.\\n",
        name);
    abort();
  }
  return sym;
}

static void uni_curl_shim_init(void) {
  uni_curl_setopt_p = (uni_curl_setopt_fn)uni_curl_shim_resolve("curl_easy_setopt");
  uni_curl_getinfo_p = (uni_curl_getinfo_fn)uni_curl_shim_resolve("curl_easy_getinfo");
}

int uni_curl_easy_setopt_ptr(void *handle, int option, void *value) {
  pthread_once(&uni_curl_init_once, uni_curl_shim_init);
  return uni_curl_setopt_p(handle, option, value);
}

int uni_curl_easy_setopt_long(void *handle, int option, long value) {
  pthread_once(&uni_curl_init_once, uni_curl_shim_init);
  return uni_curl_setopt_p(handle, option, value);
}

/* Typed long* out-parameter for CURLINFO_*_RESPONSE_CODE and other `long` infos. */
int uni_curl_easy_getinfo_long(void *handle, int info, long *value) {
  pthread_once(&uni_curl_init_once, uni_curl_shim_init);
  return uni_curl_getinfo_p(handle, info, value);
}

@xerial xerial Jul 6, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in the rebased commit — the shim now uses pthread_once around a single uni_curl_shim_init that resolves both function pointers. Thanks for the catch: my earlier "same value written twice" reasoning was wrong per C11's data-race rules, and the ADR is updated to say so.