Skip to content

Commit 0866b55

Browse files
Enrique SaurezCopilot
andcommitted
[nanvix] E: Phase 1C — build 8 Tier-1 text-codec modules as .so
Phase 1C of the .a -> .so migration (see nanvix-todo/cpython-static-to-shared-migration.md section 5). Builds on Phase 1B (#6, #7) by promoting the remaining 8 Tier-1 "text codec" stdlib extension modules from statically linked into python.elf to dlopen-loaded shared objects under lib/python3.12/lib-dynload/. Modules moved to *shared* in Modules/Setup.local generation (.nanvix/docker.py): - unicodedata: Unicode database lookups (the big one — 1.2 MB of unicode data tables). - _multibytecodec: shared CJK codec infrastructure. - _codecs_cn / _codecs_hk / _codecs_iso2022 / _codecs_jp / _codecs_kr / _codecs_tw: per-region CJK codec tables. None of the eight reference external libraries; they are pure C with embedded data tables. They link against the same -lc / runtime symbols that the rest of the Phase 1 modules use. Test coverage (.nanvix/test.py): - New phase1c_snippet imports each module, asserts it is NOT in sys.builtin_module_names, exercises one trivial API call to confirm dlopen + PyInit_<name> succeeded (unicodedata.lookup, _multibytecodec.__create_codec, _codecs_<region>.getcodec), and prints the resolved __file__ path. Phase 1A/1B probes retained. Validation on local toolchain (phase0-llfix): - All 8 new .so files produced and installed under lib-dynload/ (unicodedata 1193K, _codecs_jp 262K, _codecs_hk 168K, _codecs_cn 155K, _codecs_kr 145K, _multibytecodec 147K, _codecs_tw 115K, _codecs_iso2022 76K — total ~2.2 MB across the eight files). - nm python.elf no longer shows PyInit_<name> for any of the 8. - python.elf size: 19.18 MB (Phase 1B) -> 17.48 MB (Phase 1C), -1.70 MB. Biggest single-phase reduction so far because the CJK codec tables and the Unicode database are large. - Hello + Phase 1A + Phase 1B + Phase 1C import probes + lxml + HTTP smoke + full regrtest 160/160 PASS in standalone mode. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent b91f9d8 commit 0866b55

2 files changed

Lines changed: 31 additions & 0 deletions

File tree

.nanvix/docker.py

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -320,6 +320,15 @@ def _generate_setup_local_cmd() -> str:
320320
f"'_statistics _statisticsmodule.c' "
321321
f"'mmap mmapmodule.c' "
322322
f"'_contextvars _contextvarsmodule.c' "
323+
f"'# Phase 1C: Tier-1 text codecs (pure C, no external deps).' "
324+
f"'unicodedata unicodedata.c' "
325+
f"'_multibytecodec cjkcodecs/multibytecodec.c' "
326+
f"'_codecs_cn cjkcodecs/_codecs_cn.c' "
327+
f"'_codecs_hk cjkcodecs/_codecs_hk.c' "
328+
f"'_codecs_iso2022 cjkcodecs/_codecs_iso2022.c' "
329+
f"'_codecs_jp cjkcodecs/_codecs_jp.c' "
330+
f"'_codecs_kr cjkcodecs/_codecs_kr.c' "
331+
f"'_codecs_tw cjkcodecs/_codecs_tw.c' "
323332
f"> {ws}/Modules/Setup.local"
324333
)
325334

.nanvix/test.py

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -488,6 +488,27 @@ def stage(
488488
" print(f'CPYTHON_TEST_PHASE1A: {_name} loaded via dlopen from "
489489
"{_mod.__file__}')\n"
490490
)
491+
# Phase 1C: Tier-1 text-codec modules (CJK + unicodedata). No
492+
# external deps; resolve against python.elf's .dynsym via dlopen.
493+
phase1c_snippet = (
494+
"_phase1c = [\n"
495+
" ('unicodedata', lambda m: m.lookup('LATIN SMALL LETTER A') == 'a'),\n"
496+
" ('_multibytecodec', lambda m: hasattr(m, '__create_codec')),\n"
497+
" ('_codecs_cn', lambda m: hasattr(m, 'getcodec')),\n"
498+
" ('_codecs_hk', lambda m: hasattr(m, 'getcodec')),\n"
499+
" ('_codecs_iso2022', lambda m: hasattr(m, 'getcodec')),\n"
500+
" ('_codecs_jp', lambda m: hasattr(m, 'getcodec')),\n"
501+
" ('_codecs_kr', lambda m: hasattr(m, 'getcodec')),\n"
502+
" ('_codecs_tw', lambda m: hasattr(m, 'getcodec')),\n"
503+
"]\n"
504+
"for _name, _check in _phase1c:\n"
505+
" _mod = __import__(_name)\n"
506+
" assert _name not in sys.builtin_module_names, "
507+
"f'{_name} still built-in!'\n"
508+
" assert _check(_mod), f'{_name} sanity check failed'\n"
509+
" print(f'CPYTHON_TEST_PHASE1C: {_name} loaded via dlopen from "
510+
"{_mod.__file__}')\n"
511+
)
491512
# Phase 1B: Tier-1 math + memory modules. Same dlopen flow; libm
492513
# symbols are pulled from python.elf via --whole-archive.
493514
phase1b_snippet = (
@@ -526,6 +547,7 @@ def stage(
526547
+ array_snippet
527548
+ phase1a_snippet
528549
+ phase1b_snippet
550+
+ phase1c_snippet
529551
+ (lxml_snippet if standalone else ""),
530552
)
531553

0 commit comments

Comments
 (0)