NGWPC NWM PI-3 Formulation Selection Delivery 2026-02-25

[cmaynard-ngwpc]

This pull request delivers major stability, correctness, and capability improvements to ngen, with a strong focus on:

• BMI lifecycle correctness and memory safety
• Output variable units, conversion, and multi-formulation behavior
• Forcing engine robustness
• EWTS logging consistency
• Docker / CI modernization
• Fixes real correctness bugs (SWE units, use-after-free, MPI teardown)
• Makes units optional, resilient, and testable in multi-formulation runs
• Reduces fragility during shutdown, failure, and partial-configuration cases
• Brings documentation and tests in line with actual runtime behavior
• Submodule alignment across the NGWPC stack

A substantial portion of the work addresses use-after-free and teardown ordering bugs, introduces fully optional and resilient unit handling in multi-formulation realizations, and aligns documentation and tests with the updated behavior.

---

Additions

Output Variables & Units

• Added unit conversion support in forcing providers (fixes SWE bug).

• Added support for querying provider-native units via get_provider_units_for_variable.

• Added capability to:

  • Default to BMI native units when output units are not specified
  • Handle empty output variable lists safely
  • Gracefully fall back to raw values when unit conversion fails

• Added runtime error messages and logging when unit queries fail.

• Added unit test coverage for multi-formulation output indexing.

BMI Lifecycle & Safety

• Added explicit guarantees that BMIs are destroyed before MPI_Finalize.

• Added safeguards ensuring:

  • BMI models are not accessed after destruction
  • Forcing engine providers are finalized during instance teardown

• Added clearer runtime error reporting for invalid BMI states.

Configuration & Documentation

• Updated:

  • REALIZATION_CONFIGURATION.md
  • BMI_MODELS.md
  • Output variable documentation (units, configuration, behavior)

• Added documentation clarifying unit conversion behavior and limitations.

CI / Docker

• Added Docker support for:

  • Parameterized forcing base image names
  • Updated EWTS Python package handling

• Updated CI/CD to support ngwpc-candidate and ngwpc-release branches.

---

Changes

BMI Multi-Formulation & Output Handling

• Refactored Bmi_Multi_Formulation and Bmi_Module_Formulation to:
  • Separate timestep updates from value retrieval
  • Correctly pass time arguments to forcing vs BMI providers
  • Remove unused or unimplemented APIs
• Simplified and corrected logic for:
  • Selecting last-module outputs
  • Resolving native vs requested units
  • Writing output variable values consistently
• Fixed bugs where unconverted output values were returned incorrectly.
• Ensured unit-conversion errors are logged once per catchment, not repeatedly.

Memory Safety & Runtime Stability

• Fixed multiple use-after-free scenarios.

• Removed unsafe or redundant:

  • Explicit Python interpreter release calls
  • Null checks that masked invalid states

• Corrected teardown ordering to prevent invalid access patterns.

Logging & EWTS Integration

• Standardized logging behavior across ngen and submodules:

  • Moved GetLogLevel / IsLoggingEnabled to match other C++ modules
  • Fixed log-level initialization
  • Removed noisy or redundant log messages

• Improved exception logging when CSV forcing or initialization fails.

Forcing Engine Improvements

• Added validation for empty CSV forcing files.
• Improved error handling during formulation initialization.
• Added dynamic CAT-ID data type support.
• Ensured formatting of output streams is preserved.

Submodules & Dependency Alignment

• Updated submodule references for:
  • t-route
  • cfe
  • LASAM
  • LSTM
  • SoilFreezeThaw
  • SoilMoistureProfiles
  • Snow17
  • ueb_bmi
• Synced submodules for:
  • EWTS logging changes
  • Coastal coupling fixes
  • Official NGWPC releases

---

Removals

• Disused code that obscured model parameter investigations
• Unimplemented APIs (e.g., get_output_variable_units)
• Legacy variable-mapping checks no longer needed
• Extraneous debug logging and commented-out code
• Redundant formatting-only changes that inflated diffs.
• Unnecessary interpreter lifecycle management.