[nanvix] E: Build _ssl/_hashlib/_ctypes + lxml extensions as .so

Builds 3 stdlib extensions (_ssl, _hashlib, _ctypes) and the 2 lxml
extensions as runtime-loaded .so files, and confirms that libffi /
libssl / libcrypto / libxml2 / libxslt / libexslt / lxml ship as
standalone .so files referenced via DT_NEEDED.

This matches upstream cpython's default behavior on every Linux
distro (system-library detection enabled): cpython's normal
MODULE_*_LDFLAGS machinery wires each extension's per-module
*_LIBS env var ("-L/-l" form) into its .so link line, producing a
proper DT_NEEDED entry rather than embedding the .a.

Modules covered
---------------
- _ssl.cpython-312.so      -> DT_NEEDED libssl.so -> libcrypto.so
- _hashlib.cpython-312.so  -> DT_NEEDED libcrypto.so
- _ctypes.cpython-312.so   -> DT_NEEDED libffi.so
- _lxml_etree.cpython-312.so + _lxml_elementpath.cpython-312.so:
  thin dlopen shims (see lxml shim layer note below).

python.elf LIBS drops -lssl -lcrypto -lffi (those symbols come from
the standalone .so files now). _ssl and _hashlib previously embedded
their own copies of libcrypto.a (~5.5 MB each); the deduplication
ends the OpenSSL init-order bug where the two copies maintained
separate provider state and the first openssl_sha256() call failed.

Out of scope: _bz2 / _lzma / zlib / _sqlite3. Their Nanvix port
repos do not yet ship .so builds (the Wave 6 port-repo PRs add
those), so those four extensions stay statically linked into
python.elf for now -- the same shape upstream cpython produces
without --with-system-* flags. The follow-up cpython PR migrates
them once Wave 6 lands.

lxml shim layer
---------------
The lxml port repo (nanvix/lxml) currently emits Cython outputs as
system shared libraries (/lib/liblxml_*.so) rather than as native
cpython extension modules (site-packages/lxml/etree.cpython-312-*.so,
the upstream pip-installed layout). Until the port is fixed,
Modules/lxml_etree_builtin.c and Modules/lxml_elementpath_builtin.c
remain in place as thin dlopen shims (via the new
Modules/lxml_dlopen_shim.h helper) that load the system libraries
at runtime and forward the cpython PyInit_* symbol.

The handle is intentionally leaked on success because lxml.etree
owns process-lifetime parser state -- unloading would invalidate
live xmlNodePtr values in the Python heap. RTLD_GLOBAL lets a
hypothetical future libxml2/libxslt consumer (e.g., xmlsec) resolve
its references against the symbols this dlopen publishes.

This shim layer is a Nanvix-specific stopgap; once the lxml port
emits native cpython extensions, the entire shim layer goes away
and cpython's standard importer loads lxml.etree directly. Tracked
in nanvix-todo/lxml-port-ship-as-native-cpython-extensions.md.

Mechanics
---------
- .nanvix/setup_local.py: adds 3 *shared* entries (_ssl, _hashlib,
  _ctypes); flips _lxml_etree / _lxml_elementpath from a
  static-link form to bare-source entries (the shim now dlopens
  the .so chain at runtime).
- .nanvix/config.py: drops -lssl -lcrypto -lffi from python.elf LIBS.
- Makefile.nanvix: LIBS drops -lssl -lcrypto -lffi. Per-module
  *_LIBS env vars (LIBSQLITE3_LIBS, ZLIB_LIBS, BZIP2_LIBS,
  LIBLZMA_LIBS, LIBFFI_LIBS) stay in -L/-l form -- same as upstream
  cpython's system-library detection produces.
- .nanvix/runtime_sos.py (new): REQUIRED_RUNTIME_SOS + helper to
  stage the 8 .so files (libxml2, libxslt, libexslt, liblxml_etree,
  liblxml_elementpath, libffi, libcrypto, libssl) from buildroot/lib/
  into sysroot/lib/, with fail-fast FileNotFoundError on a missing
  dep. Consumed by both .nanvix/test.py and .nanvix/package.py.
- .nanvix/test.py: adds one row to _SO_MODULE_SANITY_CHECKS for
  _ssl / _hashlib / _ctypes.
- .nanvix/z.py: _DEP_EXPECTED_LIBS["libffi"] adds libffi.so;
  _DEP_EXPECTED_LIBS["openssl"] adds libssl.so + libcrypto.so.
- Modules/lxml_dlopen_shim.h (new): shared dlopen helper used by
  both lxml shim .c files.
- Modules/lxml_etree_builtin.c and lxml_elementpath_builtin.c:
  each shrinks to ~10 lines wrapping a single lxml_dlopen_shim() call.

Runtime dependencies
--------------------
- nanvix/nanvix#2472 -- libm visibility fix (inherited from PR-A).
- nanvix/nanvix#2473 -- dlfcn init-array + DT_RUNPATH support.

Two additional Nanvix loader fixes are required for this PR to
function at run time and must land upstream before this PR is
mergeable:
- A loader fix for diamond DT_NEEDED chains so libcrypto.so reached
  via libssl.so and directly from _hashlib.cpython-312.so resolves
  to a single mapped instance.
- A pthread_once fix in libposix so OpenSSL's lazy provider init
  does not hang on first call.

Companion port-lib repos: the Nanvix port libraries must be the
versions that produce the corresponding .so files (libffi, openssl,
libxml2, libxslt, lxml). Already pinned in cpython's nanvix.toml
and consumed via the buildroot snapshot.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: Enrique Saurez

.nanvix/config.py

@@ -109,7 +109,7 @@ def configure_env(toolchain: str | Path, sysroot: str | Path) -> dict[str, str]:
             f"{tp['libc']} {tp['libm']} "
             f"-lstdc++ -lgcc -Wl,--no-whole-archive "
             f"-Wl,--start-group "
-            f"-lsqlite3 -lssl -lcrypto -lz -lbz2 -llzma -lffi -Wl,--end-group"
+            f"-lsqlite3 -lz -lbz2 -llzma -Wl,--end-group"
         ),
         "LIBSQLITE3_LIBS": f"-L{sr}/lib -lsqlite3",
         "LIBSQLITE3_CFLAGS": f"-I{sr}/include",

.nanvix/package.py

@@ -23,6 +23,7 @@
 build_mod = load_sibling("build", __file__)
 lxml_mod = load_sibling("lxml", __file__)
 ramfs_mod = load_sibling("ramfs", __file__)
+runtime_sos_mod = load_sibling("runtime_sos", __file__)
 
 
 def _artifact_base(
@@ -166,6 +167,16 @@ def package(
     if py_lib.is_dir():
         shutil.copytree(py_lib, ramfs_sysroot / config.PYTHON_LIB_DIR)
 
+    # Stage the lxml + libxml2/libxslt/libexslt + libffi + libssl/libcrypto
+    # shared libraries from the buildroot into the release ramfs at
+    # sysroot/lib/<name>.so so the corresponding C extensions can dlopen
+    # them at runtime (see .nanvix/runtime_sos.py for the DT_NEEDED chain).
+    runtime_sos_mod.stage_runtime_sos(
+        repo_root / ".nanvix" / "buildroot" / "lib",
+        ramfs_sysroot,
+        target_label="release",
+    )
+
     ramfs_mod.trim_sysroot(ramfs_staging)
 
     # --- Include python.elf binary ---

.nanvix/runtime_sos.py

@@ -0,0 +1,78 @@
+# Copyright(c) The Maintainers of Nanvix.
+# Licensed under the MIT License.
+
+"""Runtime .so staging for Nanvix CPython.
+
+Single source of truth for the list of shared libraries that must be
+present under sysroot/lib/ at run time so the corresponding C extension
+.so files (lxml.etree, ctypes, ssl, hashlib) can dlopen them.
+
+Consumers:
+- .nanvix/test.py::stage     -- staged into the test sysroot
+- .nanvix/package.py::package -- staged into the release ramfs
+"""
+
+from __future__ import annotations
+
+import shutil
+from pathlib import Path
+from typing import Iterable
+
+# Names of .so files that must exist under .nanvix/buildroot/lib/ after
+# `./z setup` and must be staged into the test or release sysroot at
+# sysroot/lib/<name>. Failure to find any of these is fatal: the
+# corresponding C extension (lxml.etree, ctypes, ssl, hashlib) would
+# fail at first import.
+#
+# The .so files form a DT_NEEDED chain that the Nanvix dynamic loader
+# walks at dlopen time:
+#
+#   _lxml_etree.cpython-312.so       -> liblxml_etree.so
+#     liblxml_etree.so               -> libxslt.so + libexslt.so + libxml2.so
+#       libxslt.so / libexslt.so     -> libxml2.so
+#   _lxml_elementpath.cpython-312.so -> liblxml_elementpath.so
+#     liblxml_elementpath.so         -> libxml2.so
+#   _ctypes.cpython-312.so           -> libffi.so
+#   _ssl.cpython-312.so              -> libssl.so + libcrypto.so
+#     libssl.so                      -> libcrypto.so
+#   _hashlib.cpython-312.so          -> libcrypto.so
+REQUIRED_RUNTIME_SOS: tuple[str, ...] = (
+    "libxml2.so",
+    "libxslt.so",
+    "libexslt.so",
+    "liblxml_etree.so",
+    "liblxml_elementpath.so",
+    "libffi.so",
+    "libcrypto.so",
+    "libssl.so",
+)
+
+
+def stage_runtime_sos(
+    buildroot_lib: Path,
+    sysroot_lib: Path,
+    *,
+    target_label: str,
+    required: Iterable[str] = REQUIRED_RUNTIME_SOS,
+) -> None:
+    """Copy every entry in ``required`` from ``buildroot_lib`` to
+    ``sysroot_lib`` (creating the destination if needed).
+
+    Raises FileNotFoundError if any required .so is missing from
+    ``buildroot_lib``. ``target_label`` is interpolated into the error
+    message so the caller's context (test stage vs. release package) is
+    visible in the traceback.
+    """
+    sysroot_lib.mkdir(parents=True, exist_ok=True)
+    required = tuple(required)
+    missing = [name for name in required if not (buildroot_lib / name).is_file()]
+    if missing:
+        raise FileNotFoundError(
+            f"Cannot stage {target_label}: required shared libraries "
+            f"missing from {buildroot_lib}: {', '.join(missing)}. Run "
+            "`./z setup` to populate the buildroot, or rebuild the upstream "
+            "port libraries."
+        )
+    for so_name in required:
+        shutil.copy2(buildroot_lib / so_name, sysroot_lib / so_name)
+        print(f"  Staged {so_name} -> sysroot/lib/")

.nanvix/setup_local.py

@@ -91,30 +91,28 @@ class SetupEntry(NamedTuple):
             "take precedence."
         ),
     ),
+    # lxml shims: each is a thin .c that dlopens its matching .so from
+    # lib/liblxml_*.so at runtime (see Modules/lxml_dlopen_shim.h). The
+    # .so files declare DT_NEEDED entries for libxslt.so + libexslt.so +
+    # libxml2.so so the Nanvix dynamic loader walks the chain at dlopen
+    # time and python.elf no longer embeds any lxml/libxml2/libxslt code.
     SetupEntry(
         name="_lxml_etree",
         linkage=Linkage.SHARED,
-        tokens=(
-            "lxml_etree_builtin.c",
-            "-L{sysroot}/lib",
-            "-llxml_etree",
-            "-lxslt",
-            "-lexslt",
-            "-lxml2",
-            "-lz",
+        tokens=("lxml_etree_builtin.c",),
+        section_header=(
+            "lxml C extensions as dlopen-from-/lib shims. The actual "
+            "Cython code lives in liblxml_etree.so + "
+            "liblxml_elementpath.so staged into sysroot/lib/ by "
+            ".nanvix/test.py and .nanvix/package.py. No per-module "
+            "-L/-l flags here -- the loader walks the DT_NEEDED chain "
+            "at dlopen time."
         ),
-        section_header=("lxml C extension modules (linked via pre-built archives)."),
     ),
     SetupEntry(
         name="_lxml_elementpath",
         linkage=Linkage.SHARED,
-        tokens=(
-            "lxml_elementpath_builtin.c",
-            "-L{sysroot}/lib",
-            "-llxml_elementpath",
-            "-lxml2",
-            "-lz",
-        ),
+        tokens=("lxml_elementpath_builtin.c",),
     ),
     # ---------------- Data primitives (no external deps) ----------------
     *(
@@ -275,6 +273,41 @@ class SetupEntry(NamedTuple):
     ),
     SetupEntry(name="fcntl", linkage=Linkage.SHARED, tokens=("fcntlmodule.c",)),
     SetupEntry(name="termios", linkage=Linkage.SHARED, tokens=("termios.c",)),
+    # ---------------- Modules with external Nanvix-ported deps ---------
+    #
+    # libffi, libssl, libcrypto each ship as a .so under $(SYSROOT)/lib/
+    # and the consuming extension .so (_ctypes, _ssl, _hashlib)
+    # references it via DT_NEEDED. The loader resolves them at dlopen
+    # time and binds UND symbols against python.elf .dynsym.
+    #
+    # _bz2 / _lzma / zlib / _sqlite3 are intentionally NOT moved here:
+    # the Nanvix port repos for libbz2 / liblzma / libz / libsqlite3 do
+    # not yet ship .so builds, so those four extensions stay statically
+    # built into python.elf (cpython upstream default) until the
+    # follow-up PR that lands alongside the Wave 6 port-repo .so PRs.
+    SetupEntry(
+        name="_ssl",
+        linkage=Linkage.SHARED,
+        tokens=("_ssl.c",),
+        section_header=(
+            "Stdlib modules with external Nanvix-ported deps that are "
+            "already shipped as .so by their respective port repos. "
+            "Each .so emits DT_NEEDED for the corresponding sysroot "
+            "library; the loader walks the chain at dlopen time."
+        ),
+    ),
+    SetupEntry(name="_hashlib", linkage=Linkage.SHARED, tokens=("_hashopenssl.c",)),
+    SetupEntry(
+        name="_ctypes",
+        linkage=Linkage.SHARED,
+        tokens=(
+            "_ctypes/_ctypes.c",
+            "_ctypes/callbacks.c",
+            "_ctypes/callproc.c",
+            "_ctypes/stgdict.c",
+            "_ctypes/cfield.c",
+        ),
+    ),
 )
 
 

.nanvix/test.py

@@ -33,6 +33,7 @@
 build_mod = load_sibling("build", __file__)
 lxml_mod = load_sibling("lxml", __file__)
 ramfs_mod = load_sibling("ramfs", __file__)
+runtime_sos_mod = load_sibling("runtime_sos", __file__)
 
 
 # ---------------------------------------------------------------------------
@@ -105,6 +106,16 @@
             ("termios", "hasattr(m, 'tcgetattr')"),
         ),
     ),
+    (
+        "CPYTHON_TEST_EXTERNAL_DEPS",
+        (
+            # libffi / libssl / libcrypto: each consumed via DT_NEEDED of
+            # the corresponding sysroot/lib/<name>.so.
+            ("_ssl", "hasattr(m, 'RAND_bytes')"),
+            ("_hashlib", "hasattr(m, 'openssl_sha256') or hasattr(m, 'new')"),
+            ("_ctypes", "hasattr(m, 'dlopen')"),
+        ),
+    ),
 )
 
 
@@ -522,6 +533,16 @@ def stage(
 
     sysroot_dir = staging / "sysroot"
 
+    # Stage the lxml/libxml2/libxslt/libexslt + libffi + libssl/libcrypto
+    # shared libraries from the buildroot into the test sysroot at
+    # sysroot/lib/<name>.so so the corresponding C extensions can dlopen
+    # them at runtime (see .nanvix/runtime_sos.py for the DT_NEEDED chain).
+    runtime_sos_mod.stage_runtime_sos(
+        repo_root / ".nanvix" / "buildroot" / "lib",
+        sysroot_dir / "lib",
+        target_label="test sysroot",
+    )
+
     # Ensure _sysconfigdata module is present in the installed sysroot.
     # make install should copy it from build/<pybuilddir>/ but this can
     # silently fail when PYTHON_FOR_BUILD is not available or when the

.nanvix/z.py

@@ -80,10 +80,10 @@
 # Map dependency names to the library files they install into buildroot/lib.
 _DEP_EXPECTED_LIBS: dict[str, list[str]] = {
     "bzip2": ["libbz2.a"],
-    "libffi": ["libffi.a"],
+    "libffi": ["libffi.a", "libffi.so"],
     "zlib": ["libz.a"],
     "sqlite": ["libsqlite3.a"],
-    "openssl": ["libssl.a", "libcrypto.a"],
+    "openssl": ["libssl.a", "libcrypto.a", "libssl.so", "libcrypto.so"],
     "libxml2": ["libxml2.a"],
     "libxslt": ["libxslt.a", "libexslt.a"],
     "lxml": ["liblxml_etree.a", "liblxml_elementpath.a"],

Makefile.nanvix

@@ -185,11 +185,17 @@ endif
 # which libnvx_crt0 is the sole provider. Listing it first today keeps
 # python.elf using a consistent `_start` source across both states.
 #
-# `LIBS` segment 2 (`--start-group ... --end-group`): the external add-on
-# libraries (sqlite3, ssl, crypto, z, bz2, lzma, ffi). The group is needed
-# only for their inter-archive circular dependencies; they resolve symbols
-# from libposix/libc/libm/libstdc++ against the already-embedded objects
-# from segment 1.
+# `LIBS` segment 2 (`--start-group ... --end-group`): the external
+# Nanvix-ported libraries (`-lsqlite3 -lz -lbz2 -llzma`) whose .a files
+# stay statically linked into python.elf. The corresponding extensions
+# (_sqlite3, zlib, _bz2, _lzma) remain *static* in Setup.local for now
+# because the Nanvix port repos for libbz2 / libz / liblzma /
+# libsqlite3 do not yet ship .so builds. The follow-up cpython PR that
+# lands alongside the Wave 6 port-repo .so PRs migrates these four
+# extensions to *shared* with DT_NEEDED references to lib*.so. -lssl
+# / -lcrypto / -lffi are NOT in this list -- they ship as separate .so
+# files under sysroot/lib/ and are referenced via DT_NEEDED by their
+# consumer .so modules (_ssl, _hashlib, _ctypes).
 CONFIGURE_ENV = \
 	CC="$(TOOLCHAIN_PREFIX)/bin/i686-nanvix-gcc" \
 	CXX="$(TOOLCHAIN_PREFIX)/bin/i686-nanvix-g++" \
@@ -199,7 +205,7 @@ CONFIGURE_ENV = \
 	CFLAGS="-O3 -fomit-frame-pointer -fno-unwind-tables -fno-asynchronous-unwind-tables -I$(SYSROOT_PATH)/include" \
 	CFLAGS_NODIST="-fno-semantic-interposition" \
 	LDFLAGS="-L$(SYSROOT_PATH)/lib -T$(SYSROOT_PATH)/lib/user.ld -Wl,--allow-multiple-definition -no-pie -Wl,--export-dynamic -Wl,--no-dynamic-linker" \
-	LIBS="-Wl,--whole-archive $(LIBNVX_CRT0) $(LIBPOSIX) $(LIBC) $(LIBM) $(LIBSTDCXX) $(LIBGCC) -Wl,--no-whole-archive -Wl,--start-group -lsqlite3 -lssl -lcrypto -lz -lbz2 -llzma -lffi -Wl,--end-group" \
+	LIBS="-Wl,--whole-archive $(LIBNVX_CRT0) $(LIBPOSIX) $(LIBC) $(LIBM) $(LIBSTDCXX) $(LIBGCC) -Wl,--no-whole-archive -Wl,--start-group -lsqlite3 -lz -lbz2 -llzma -Wl,--end-group" \
 	LIBSQLITE3_LIBS="-L$(SYSROOT_PATH)/lib -lsqlite3" \
 	LIBSQLITE3_CFLAGS="-I$(SYSROOT_PATH)/include" \
 	ZLIB_LIBS="-L$(SYSROOT_PATH)/lib -lz" \

Modules/lxml_dlopen_shim.h

@@ -0,0 +1,65 @@
+/*
+ * lxml_dlopen_shim.h -- shared helper for the two lxml dlopen shims
+ *                       (lxml_etree_builtin.c, lxml_elementpath_builtin.c).
+ *
+ * makesetup does not support dotted module names, so the Cython
+ * extensions are registered under the flat names "_lxml_etree" /
+ * "_lxml_elementpath" via Modules/Setup.local and bridged back to
+ * lxml.etree / lxml._elementpath by a pure-Python shim at lxml/etree.py
+ * (see .nanvix/lxml.py::_ETREE_SHIM).
+ *
+ * On Nanvix the Cython-generated code lives in liblxml_etree.so /
+ * liblxml_elementpath.so under sysroot/lib/, each of which declares
+ * DT_NEEDED entries for libxslt.so + libexslt.so + libxml2.so (etree)
+ * or libxml2.so (elementpath). Loading those .so files via dlopen lets
+ * the Nanvix dynamic loader walk that chain at runtime, so each
+ * library lives at one address in memory and python.elf does not have
+ * to embed multiple megabytes of libxml2/libxslt code statically.
+ *
+ * The dlopen handle is INTENTIONALLY LEAKED on success: lxml.etree owns
+ * process-lifetime parser state (xmlInitParser globals), so unloading
+ * the library would invalidate still-live xmlNodePtr values in the
+ * Python heap.
+ *
+ * RTLD_GLOBAL so subsequently loaded extensions that link back into
+ * libxml2/libxslt (e.g., a future xmlsec) can resolve their undefined
+ * references against the symbols this dlopen publishes.
+ */
+
+#ifndef NANVIX_LXML_DLOPEN_SHIM_H
+#define NANVIX_LXML_DLOPEN_SHIM_H
+
+#include "Python.h"
+
+#include <dlfcn.h>
+
+typedef PyObject *(*lxml_dlopen_init_fn)(void);
+
+static inline PyObject *
+lxml_dlopen_shim(const char *so_path, const char *init_symbol)
+{
+    void *handle = dlopen(so_path, RTLD_NOW | RTLD_GLOBAL);
+    if (handle == NULL) {
+        PyErr_Format(PyExc_ImportError,
+                     "dlopen(\"%s\") failed: %s", so_path, dlerror());
+        return NULL;
+    }
+
+    lxml_dlopen_init_fn init =
+        (lxml_dlopen_init_fn)dlsym(handle, init_symbol);
+    if (init == NULL) {
+        const char *err = dlerror();
+        PyErr_Format(PyExc_ImportError,
+                     "dlsym(\"%s\") in %s failed: %s",
+                     init_symbol, so_path,
+                     err ? err : "symbol not found");
+        /* Release the handle on this error path. The success path
+         * intentionally leaks it -- see file header. */
+        dlclose(handle);
+        return NULL;
+    }
+
+    return init();
+}
+
+#endif /* NANVIX_LXML_DLOPEN_SHIM_H */

Modules/lxml_elementpath_builtin.c

@@ -1,16 +1,15 @@
 /*
- * lxml_elementpath_builtin.c - Shim for lxml._elementpath built-in module.
+ * lxml_elementpath_builtin.c -- Nanvix dlopen shim for lxml._elementpath.
  *
- * Registers the Cython _elementpath extension under the flat name
- * "_lxml_elementpath" for Modules/Setup.local.
+ * See lxml_dlopen_shim.h for the design rationale.
  */
 
-#include "Python.h"
+#include "lxml_dlopen_shim.h"
 
-extern PyObject* PyInit__elementpath(void);
+#define LXML_ELEMENTPATH_SO "lib/liblxml_elementpath.so"
 
 PyMODINIT_FUNC
 PyInit__lxml_elementpath(void)
 {
-    return PyInit__elementpath();
+    return lxml_dlopen_shim(LXML_ELEMENTPATH_SO, "PyInit__elementpath");
 }