Skip to content

fix: Resolve libcurl symbols via dlsym so Scala Native binaries without CurlBindings link (#622) - #636

Merged
xerial merged 1 commit into
mainfrom
fix/curl-shim-link-error-622
Jul 6, 2026
Merged

fix: Resolve libcurl symbols via dlsym so Scala Native binaries without CurlBindings link (#622)#636
xerial merged 1 commit into
mainfrom
fix/curl-shim-link-error-622

Conversation

@xerial

@xerial xerial commented Jul 6, 2026

Copy link
Copy Markdown
Member

Summary

Fixes #622. Since v2026.1.13, Scala Native builds depending on uni-native fail to link with undefined reference to curl_easy_setopt whenever downstream code doesn't touch CurlBindings (e.g. a project that only uses LogSupport). Scala Native compiles resources/scala-native/*.c from every jar on the classpath unconditionally; the shim's strong extern references to libcurl symbols couldn't be satisfied because -lcurl only reaches the linker via @link("curl") on CurlBindings.Extern, which DCE strips when unused.

  • uni_curl_shim.c now resolves curl_easy_setopt / curl_easy_getinfo lazily via dlsym(RTLD_DEFAULT, ...) inside a pthread_once init, instead of C-level externs. The compiled .o carries no libcurl symbol references, so downstream links succeed regardless of whether libcurl is on the linker command line.
  • Variadic ABI (the reason the shim exists — fix: Repair the Native libcurl client (CURLE_URL_MALFORMAT via C shim) #580) is preserved by declaring the function-pointer typedefs variadic, so the C compiler still emits the correct calling convention at the shim's call sites.
  • #define _GNU_SOURCE before <dlfcn.h> so RTLD_DEFAULT is exposed on glibc; harmless on macOS.
  • __attribute__((weak)) on the extern was tried first — works on Linux, fails on macOS because ld still needs -undefined dynamic_lookup for weak-undefineds, and a jar-resource .c cannot force that flag on downstream builds.

Design decision documented in adr/2026-07-06-curl-shim-weak-linking.md.

Test plan

  • Standalone: clang -c uni_curl_shim.c — the resulting .o has zero libcurl symbol references (nm -m shows only _abort, _dlsym, _fprintf, _pthread_once, ___stderrp undefined).
  • Standalone: link a program that doesn't reference libcurl against the shim .o — succeeds (simulates the ScalaNative - unable to build when using uni::2026.1.13 (works with uni::2026.1.12 and earlier) #622 downstream scenario).
  • Scala-Native-produced .o (from target/out/native0.5/.../uni_curl_shim.c.o) also shows no libcurl externals.
  • ./sbt "uniNative/testOnly *NativeCurlClientTest" — 3/3 pass (variadic ABI still correct end-to-end).
  • ./sbt "projectNative/test" — 327/327 pass (no regressions).
  • End-to-end reproduction of the reporter's scala-cli run . --native case — to be validated post-release.

@github-actions github-actions Bot added the bug Something isn't working label Jul 6, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses a linking issue in downstream Scala Native projects (issue #622) by modifying uni_curl_shim.c to dynamically resolve libcurl symbols (curl_easy_setopt and curl_easy_getinfo) via dlsym instead of using static extern declarations, and documents this change in a new Architecture Decision Record (ADR). The review feedback points out that the lazy initialization of the function pointers is not thread-safe and could lead to undefined behavior, suggesting pthread_once as a solution. Additionally, defining _GNU_SOURCE is recommended to ensure the portability of RTLD_DEFAULT on Linux systems.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +51 to 85
static uni_curl_setopt_fn uni_curl_setopt_p = NULL;
static uni_curl_getinfo_fn uni_curl_getinfo_p = NULL;

static void *uni_curl_shim_resolve(const char *name) {
void *sym = dlsym(RTLD_DEFAULT, name);
if (sym == NULL) {
fprintf(
stderr,
"uni_curl_shim: libcurl symbol '%s' not found in this process. Add "
"wvlet.uni.http.CurlBindings to your Scala Native build (which links "
"libcurl via @link(\"curl\")), or install libcurl. See wvlet/uni#622.\n",
name);
abort();
}
return sym;
}

int uni_curl_easy_setopt_ptr(void *handle, int option, void *value) {
return curl_easy_setopt(handle, option, value);
if (uni_curl_setopt_p == NULL)
uni_curl_setopt_p = (uni_curl_setopt_fn)uni_curl_shim_resolve("curl_easy_setopt");
return uni_curl_setopt_p(handle, option, value);
}

int uni_curl_easy_setopt_long(void *handle, int option, long value) {
return curl_easy_setopt(handle, option, value);
if (uni_curl_setopt_p == NULL)
uni_curl_setopt_p = (uni_curl_setopt_fn)uni_curl_shim_resolve("curl_easy_setopt");
return uni_curl_setopt_p(handle, option, value);
}

/* Typed long* out-parameter for CURLINFO_*_RESPONSE_CODE and other `long` infos. */
int uni_curl_easy_getinfo_long(void *handle, int info, long *value) {
return curl_easy_getinfo(handle, info, value);
if (uni_curl_getinfo_p == NULL)
uni_curl_getinfo_p = (uni_curl_getinfo_fn)uni_curl_shim_resolve("curl_easy_getinfo");
return uni_curl_getinfo_p(handle, info, value);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The current lazy initialization of uni_curl_setopt_p and uni_curl_getinfo_p is not thread-safe. If multiple threads concurrently invoke the shim functions when the pointers are NULL, they will concurrently read and write to these static variables. In C11, concurrent unsynchronized accesses to non-atomic variables where at least one is a write constitutes a data race, which is undefined behavior (UB). Even if they write the same value, compiler optimizations can lead to unexpected behavior.

To resolve this portably and efficiently without adding complex locks or relying on compiler-specific atomics, use POSIX's pthread_once. This guarantees thread-safe, one-time initialization of both function pointers.

static uni_curl_setopt_fn  uni_curl_setopt_p  = NULL;
static uni_curl_getinfo_fn uni_curl_getinfo_p = NULL;
static pthread_once_t uni_curl_init_once = PTHREAD_ONCE_INIT;

static void *uni_curl_shim_resolve(const char *name) {
  void *sym = dlsym(RTLD_DEFAULT, name);
  if (sym == NULL) {
    fprintf(
        stderr,
        "uni_curl_shim: libcurl symbol '%s' not found in this process. Add "
        "wvlet.uni.http.CurlBindings to your Scala Native build (which links "
        "libcurl via @link(\\\"curl\\\")), or install libcurl. See wvlet/uni#622.\\n",
        name);
    abort();
  }
  return sym;
}

static void uni_curl_shim_init(void) {
  uni_curl_setopt_p = (uni_curl_setopt_fn)uni_curl_shim_resolve("curl_easy_setopt");
  uni_curl_getinfo_p = (uni_curl_getinfo_fn)uni_curl_shim_resolve("curl_easy_getinfo");
}

int uni_curl_easy_setopt_ptr(void *handle, int option, void *value) {
  pthread_once(&uni_curl_init_once, uni_curl_shim_init);
  return uni_curl_setopt_p(handle, option, value);
}

int uni_curl_easy_setopt_long(void *handle, int option, long value) {
  pthread_once(&uni_curl_init_once, uni_curl_shim_init);
  return uni_curl_setopt_p(handle, option, value);
}

/* Typed long* out-parameter for CURLINFO_*_RESPONSE_CODE and other `long` infos. */
int uni_curl_easy_getinfo_long(void *handle, int info, long *value) {
  pthread_once(&uni_curl_init_once, uni_curl_shim_init);
  return uni_curl_getinfo_p(handle, info, value);
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in the rebased commit — the shim now uses pthread_once around a single uni_curl_shim_init that resolves both function pointers. Thanks for the catch: my earlier "same value written twice" reasoning was wrong per C11's data-race rules, and the ADR is updated to say so.

Comment on lines +40 to +42
#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

On Linux systems using glibc, the RTLD_DEFAULT constant is a GNU extension and is only exposed in <dlfcn.h> if the _GNU_SOURCE feature test macro is defined before including the header. To ensure portable compilation across all Linux distributions and toolchains, define _GNU_SOURCE at the very top of the file.

#define _GNU_SOURCE
#include <dlfcn.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed — #define _GNU_SOURCE is now the first line of the file, before <dlfcn.h>. On macOS libSystem defines RTLD_DEFAULT unconditionally so the macro is harmless there.

…tive binaries that don't use CurlBindings still link (#622)

Since v2026.1.13, Scala Native builds that depended on uni-native failed to
link with `undefined reference to curl_easy_setopt` whenever the downstream
code did not touch `CurlBindings` — Scala Native compiles `resources/scala-
native/*.c` unconditionally, and the shim's strong `extern` references to
libcurl symbols could not be satisfied because `-lcurl` only reaches the
linker via `@link("curl")` on `CurlBindings.Extern`, which DCE strips.

Resolve `curl_easy_setopt`/`curl_easy_getinfo` lazily via
`dlsym(RTLD_DEFAULT, ...)` instead of declaring them as C-level `extern`s,
so the compiled `.o` carries no libcurl symbol references. Variadic ABI
(the whole reason the shim exists — #580) is preserved by declaring the
function-pointer typedefs variadic.

Weak-`extern` was tried first but is unusable — macOS `ld` still requires
`-undefined dynamic_lookup` to leave weak-undefineds unresolved at final
link, and a jar-resource `.c` cannot force that flag on downstream builds.

See adr/2026-07-06-curl-shim-weak-linking.md.
@xerial
xerial force-pushed the fix/curl-shim-link-error-622 branch from 7a4d67a to 34667cb Compare July 6, 2026 21:53
@xerial
xerial merged commit 77510b6 into main Jul 6, 2026
15 checks passed
@xerial
xerial deleted the fix/curl-shim-link-error-622 branch July 6, 2026 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ScalaNative - unable to build when using uni::2026.1.13 (works with uni::2026.1.12 and earlier)

1 participant