Skip to content

[posix] E: Implement pthread_once and accept pthread_key_create destructors#29

Closed
esaurez wants to merge 2 commits into
feat/dlfcn-diamond-neededfrom
feat/pthread_once-and-key-destructor
Closed

[posix] E: Implement pthread_once and accept pthread_key_create destructors#29
esaurez wants to merge 2 commits into
feat/dlfcn-diamond-neededfrom
feat/pthread_once-and-key-destructor

Conversation

@esaurez

@esaurez esaurez commented Jun 6, 2026

Copy link
Copy Markdown
Owner

Summary

Implement two related libposix gaps that together block any OpenSSL 3 consumer (Python's _ssl / _hashlib, openssl CLI tooling, anything that goes through EVP_* / SSL_*):

  1. pthread_once — was a no-op stub (this PR resolves [posix] Support pthread_once() nanvix/nanvix#513) that returned 0 without ever invoking init_routine. Every consumer that uses pthread_once for lazy initialization (OpenSSL, libstdc++ static init, glib, ICU, …) was silently never initialised.

  2. pthread_key_create — rejected non-NULL destructors with OperationNotSupported. OpenSSL's ossl_init_thread always passes a destructor (defensive practice) so the call fails. With pthread_once fixed, this was the very next step in OpenSSL's init chain to break.

Why this matters now

OpenSSL 3 routes every internal one-time init through pthread_once (crypto/threads_pthread.c:684RUN_ONCE macro at include/internal/thread_once.h:129):

  • OPENSSL_init_crypto returns 0
  • Provider registry never populated
  • EVP_MD_fetch("SHA2-256") returns NULL
  • SSL_CTX_new() returns NULL with empty error queue
  • Python sees _hashlib.UnsupportedDigestmodError: unsupported hash type sha256 and ssl.SSLError: unknown error (_ssl.c:3076)

This was the long-documented "OpenSSL init-order quirk" from cpython Phase 3 (esaurez/cpython#10) where the prediction was "the Group B unbundling will fix it for free". The prediction was correct in spirit but the unbundling on its own surfaces the real underlying cause: the Nanvix pthread_once stub.

Implementation

pthread_oncesrc/libs/posix/src/pthread/mod.rs

4-state machine encoded in init_executed, mirroring musl libc's encoding so the future MT upgrade is a mechanical rewrite (swap read_volatile/write_volatile for AtomicI32::load(Acquire)/store(Release) + futex wait/wake):

Value State
0 NEVER_RUN (matches PTHREAD_ONCE_INIT)
1 DONE
2 IN_PROGRESS
3 WAIT (reserved for future MT — unused in ST)

Key properties:

  • Fast path: volatile read of init_executed; if DONE, acquire compiler fence + return. No-op on single-CPU runtime, required when SMP lands.
  • Slow path: transition NEVER_RUNIN_PROGRESS, run init_routine, transition IN_PROGRESSDONE with a release fence.
  • Cancellation safety: a Rust Drop guard resets to NEVER_RUN if init_routine panics/unwinds — POSIX-compliant ("the effect on once_control shall be as if pthread_once() was never called").
  • Sanity check: is_initialized != 1EINVAL (catches PTHREAD_ONCE_INIT misuse).
  • Recursive same-control calls (POSIX undefined): logged, returns 0 without re-running. Sensible default over deadlock; matches the contract of "don't re-run".

The 4-state encoding is documented inline; the multi-threaded upgrade path is sketched in a comment block. The struct already has init_executed, which doubles as the futex word for the upgrade.

pthread_key_createsrc/libs/syscall/src/pthread/bindings/pthread_key_create.rs

Drop the destructor-is-some hard error. Log a one-line warning and continue. Worst case is a per-thread resource leak at thread exit, which is benign in the single-threaded user-space model. Once thread-exit destructor support lands, the warning will get removed.

pthread_once_t accessors — src/libs/sysapi/src/sys_types.rs

Added pub fn is_initialized(), init_executed(), set_init_executed(), init_executed_ptr() (the last for the volatile / future-atomic state-machine word). The struct fields stay private; the accessors let libposix::pthread_once operate on the state-machine word without exposing internal layout.

Validation

Standalone C probe (statically linked OpenSSL 3.5.0 + libposix)

Before:

PROBE: OPENSSL_init_crypto=0
PROBE: OSSL_PROVIDER_load=0x0
PROBE: EVP_MD_fetch=0x0
PROBE_FAIL

After:

PROBE: OPENSSL_init_crypto=1
PROBE: OSSL_PROVIDER_load=0x6001fa70
PROBE: EVP_MD_fetch=0x60031d70
PROBE_PASS

Python end-to-end on cpython.elf

STEP_1:python_started (3, 12, 3)
STEP_2:_hashlib imported /lib/python3.12/lib-dynload/_hashlib.cpython-312.so
STEP_3:sha256 digest=a7f9173ca23c3e497cdc2e9cb1e7a238c59f2c02dc7e5a17f9dadc8860d801a5
STEP_4:_ssl imported /lib/python3.12/lib-dynload/_ssl.cpython-312.so
STEP_5:SSLContext created proto=TLS_CLIENT
STEP_6:openssl_sha256 direct digest=2d711642b726b044...
STEP_7:_hashlib and hashlib agree
OPENSSL_CHAIN_PASS

STEP_6 fixes the long-documented init-order bug where _hashlib.openssl_sha256(b"x") called as the first OpenSSL API in the process returned UnsupportedDigestmodError.

Regression tests

What's not in scope (deliberately)

  • Real thread-exit destructor invocation for pthread_key_create. Out of scope for this PR; the warning will get removed when implemented.
  • Multi-threaded pthread_once. The single-threaded variant is what current Nanvix needs; the comment block in the code documents the mechanical upgrade. Doing the upgrade requires futex_wait/futex_wake (or equivalent) in libposix, which doesn't exist yet.
  • #[repr(C, packed)] on pthread_once_t — flagged in the research as a future MT correctness concern (atomic accesses require alignment). Doesn't matter for the ST single-CPU implementation, but should be revisited at the MT upgrade. Left as-is to not change the libc ABI.

Closes

Downstream consumers

Both downstream PRs are independently complete and will be filed once this lands.

Enrique Saurez and others added 2 commits June 8, 2026 16:28
The dynamic loader walks `DT_NEEDED` recursively, but two bugs broke
diamond dependency graphs of the shape:

  libdiamond.so -> libleft.so  -> libbase.so
                -> libright.so -> libbase.so

both of which only surface once `DT_NEEDED` chains are nontrivial
(the existing dlfcn-needed-c test stays a single-edge linear graph,
so it never exercised either path). Symptoms before this commit:

  1. `dlopen(libdiamond.so)` HANGS inside the recursive load.
  2. After the hang fix is in place, `dlclose(libdiamond.so)` panics
     with `assertion left == right failed: expected to remove exactly
     one dynamic library file`.

Fix 1 -- dlopen deadlock on ancestor lock
-----------------------------------------

`load_all_dependencies_recursive` holds the parent library's mutex
while iterating the registry's already-loaded entries. Each entry
gets `.lock()`-ed to compare names. The previous code skipped only
the immediate `new_dlhandle` self; it did not skip ancestors held
by outer recursive frames. Loading libright (depth 1) and then
recursing into libright's deps (depth 2) tried to lock libdiamond
in the inner scan, which deadlocked: libdiamond was held by the
outer frame.

Fix: thread a `BTreeSet<DlHandle>` of ancestor handles down the
recursion and skip every entry whose handle is in that set. The
ancestor is by definition not a candidate for "already loaded by
a sibling" so the skip is functionally safe.

Fix 2 -- dlopen now also handles transitive load races
------------------------------------------------------

Once the deadlock is gone, a second issue surfaces: the pre-loop
`retain` only sees libraries loaded *before* the current frame
started. In a diamond, libbase is loaded by libleft's recursion
*during* libdiamond's frame, after libdiamond's retain has already
run. Without a re-check, the outer loop would then open libbase a
second time (creating two distinct in-memory copies with their own
globals) or trip the `unreachable!()` when the VFS reuses the file
descriptor.

Fix: before each `DynamicLibrary::open` call, re-scan the registry
for a matching name and bind the existing copy if found.

Fix 3 -- dlclose strong_count assertion for diamond
---------------------------------------------------

The BFS unload loop extracts each library from the registry when
`Arc::strong_count == 2` (registry + current pop). For a diamond,
libbase has three Arc references at the time it is first popped
(registry + libleft.dependencies + libright.dependencies), so
`extract_if` correctly returns 0 entries -- but the assertion
demanded exactly 1, panicking the kernel. A later iteration of the
loop pops libright/libleft, drops their dependencies (releasing
the extra libbase ref), pushes libbase again, and the second pop
succeeds at the now-correct refcount.

Fix: replace the strict `assert_eq!(== 1)` with an early `continue`
when `extract_if` returns zero entries, keeping the (now relaxed)
"at most one" assertion to catch any future runtime invariant
violation.

Validation
----------

Companion test suite at esaurez/posix-tests `dlfcn-diamond-c`
proves both fixes end-to-end on a standalone Nanvix VM:

  === dlfcn diamond DT_NEEDED tests ===
    PASS: dlopen(libright.so) [depth-2 chain]
    PASS: dlopen(libdiamond.so)
    PASS: re-dlopen(libdiamond.so) returns same handle
  3 passed, 0 failed

All previously passing dlfcn tests (dlfcn-c, dlfcn-init-runpath-c,
dlfcn-pie-c) still pass; full posix-tests suite passes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…uctors

Two related libposix gaps that together block any OpenSSL 3 consumer:

1. pthread_once was a no-op stub (nanvix#513) that returned 0 without ever invoking init_routine. Every consumer that uses pthread_once for lazy initialization (OpenSSL, libstdc++ static init, glib, ...) silently never initialized. OpenSSL 3 routes every internal one-time init through this mechanism; ossl_init_base never runs, EVP_MD_fetch returns NULL, _hashlib.openssl_sha256() raises UnsupportedDigestmodError, SSL_CTX_new returns NULL with empty error queue.

2. pthread_key_create rejected non-null destructors with OperationNotSupported. OpenSSL's ossl_init_thread always passes a destructor (defensive practice) so the call fails. With pthread_once fixed, this is the very next step in OpenSSL's init chain.

Mechanics:

- pthread_once: 4-state machine encoded in init_executed (NEVER_RUN=0, DONE=1, IN_PROGRESS=2, WAIT=3) -- musl's encoding, future-compatible with the multi-threaded futex upgrade documented in the source comments. Uses volatile read/write + compiler_fence for memory ordering. Rust Drop guard provides POSIX cancellation semantics (reset to NEVER_RUN on panic/unwind). Sanity-checks is_initialized==1 to catch PTHREAD_ONCE_INIT misuse. Recursive same-control calls log a warning and return 0 (POSIX undefined behavior; our choice over deadlock).

- pthread_key_create: drop the destructor-is-some hard error. Log a one-line warning and continue. Worst case is a per-thread resource leak at thread exit, which is benign in the single-threaded user-space model. Once thread-exit destructor support lands, the warning will get removed.

- pthread_once_t accessor methods (init_executed_ptr, is_initialized, etc.) added to sysapi so libposix can read/write the state-machine word without making the struct fields pub.

Validation:

- OpenSSL 3.5.0 standalone C probe goes PROBE_FAIL -> PROBE_PASS (EVP_MD_fetch and OSSL_PROVIDER_load both succeed).

- Python smoke tests on cpython.elf:

    OPENSSL_CHAIN_PASS (SSLContext, _hashlib.openssl_sha256 direct call -- the Phase-3 documented init-order bug is fixed)

    LXML_CHAIN_PASS (regression)

    CTYPES_CHAIN_PASS (regression)

Closes nanvix#513.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@esaurez esaurez force-pushed the feat/pthread_once-and-key-destructor branch from 8f1d424 to 78dd76d Compare June 8, 2026 23:29
@esaurez esaurez changed the base branch from dev to feat/dlfcn-diamond-needed June 8, 2026 23:29
@esaurez esaurez force-pushed the feat/dlfcn-diamond-needed branch from bfdd573 to 6f4e0ae Compare June 9, 2026 01:52
@esaurez

esaurez commented Jun 9, 2026

Copy link
Copy Markdown
Owner Author

Superseded by upstream PR nanvix#2481, which carries the same diff (cherry-picked onto upstream/dev, deep-reviewed, with esaurez/* and cpython-migration references stripped). Two SHOULD-FIX findings from the deep review were applied: (a) eliminated &mut pthread_once_t materialisation to avoid Stacked-Borrows UB on the recursive-call path the implementation explicitly supports (raw-pointer accessors added to sysapi); (b) rewrote OnceGuard comment to honestly describe its forward-compat scope (no panic-unwind across extern C, no pthread_cancel today, panic=abort in release). Closing this fork PR; tracking continues upstream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[posix] Support pthread_once()

1 participant