Skip to content

Fix OpenMP reproducibility#1480

Open
DusanJovic-NOAA wants to merge 11 commits into
MPAS-Dev:developfrom
DusanJovic-NOAA:dev_fix_omp_reproducibility
Open

Fix OpenMP reproducibility#1480
DusanJovic-NOAA wants to merge 11 commits into
MPAS-Dev:developfrom
DusanJovic-NOAA:dev_fix_omp_reproducibility

Conversation

@DusanJovic-NOAA

Copy link
Copy Markdown

Fix the reproducibility of model outputs when a threaded build runs on a different number of OpenMP threads.

I used thread sanitizer to find potential data races, which frequently cause model non-reproducibility. Not every data race results in non-reproducible output, for example, a data race where multiple threads write the same initial value to a shared memory location. I think those cases should be fixed as well to minimize the number of warnings the thread sanitizer reports.

I tested these code changes using Intel (2025.3.0) and GNU (11.5.0). To get reproducible results with optimized code (-O3) for the Intel compiler I had to use the -fp-model precise compiler flag, in addition to the flags currently specified in the CMake files. For the GNU compiler I had to use -fno-tree-loop-vectorize. I didn't add those flags to the CMake files because I haven't checked the performance impact.

With the code changes included in this PR and the compiler flags mentioned above, I am getting reproducible outputs (history, diag and restart files, checked by running nccmp) when I use 'mesoscale_reference' physics. For 'convection_permitting' physics two additional changes are needed in physics_mmm:

diff --git a/bl_mynn.F90 b/bl_mynn.F90
index 783b996..c083888 100644
--- a/bl_mynn.F90
+++ b/bl_mynn.F90
@@ -312,7 +312,7 @@
  integer,intent(out):: &
     errflg        ! output error flag (-).
 
- real(kind=kind_phys),intent(out),dimension(:):: &
+ real(kind=kind_phys),intent(out),dimension(its:):: &
     maxwidth,   &!
     maxmf,      &!
     ztop_plume

and

diff --git a/cu_ntiedtke.F90 b/cu_ntiedtke.F90
index e1d266d..75367e5 100644
--- a/cu_ntiedtke.F90
+++ b/cu_ntiedtke.F90
@@ -13,6 +13,7 @@
  real(kind=kind_phys):: g
  real(kind=kind_phys):: rd
  real(kind=kind_phys):: rv
+ !$omp threadprivate(alf, als, alv, cpd, g, rd, rv)
 
  real(kind=kind_phys),parameter:: t13   = 1.0/3.0
  real(kind=kind_phys),parameter:: tmelt = 273.16
@@ -46,6 +47,7 @@
  real(kind=kind_phys):: ralfdcp
  real(kind=kind_phys):: vtmpc1
  real(kind=kind_phys):: zrg
+ !$omp threadprivate(rcpd, c2es, c5les, c5ies, r5alvcp, r5alscp, ralvdcp, ralsdcp, ralfdcp, vtmpc1, zrg)
 
  logical,parameter:: nonequil = .true.
  logical,parameter:: lmfpen   = .true.

ThreadSanitizer warns about a potential data race. Two variables (aer_opt and ysu_pblmix)
are defined in mpas_atmphys_vars module and are global (a single copy shared by all threads).
In this specific case all threads assign 0 to both variables, which means even if
there's a data race the result will be the same regardless of the thread count.

But I think this should be fixed if nothing else then to have clean(er) sanitizer report,
which will make it easier to spot the actual data races later on.

0: ==================
0: WARNING: ThreadSanitizer: data race (pid=3577273)
0:   Write of size 4 at 0x000005375c28 by thread T2:
0:     #0 mpas_atmphys_driver_radiation_sw::radiation_sw_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:471:2 (mpas_atmosphere+0xc805f7) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#1 mpas_atmphys_driver_radiation_sw::driver_radiation_sw_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:877:7 (mpas_atmosphere+0xca19f1) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.102086.split2131 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:231:16 (mpas_atmosphere+0xa99c86) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0:     MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:229:7 (mpas_atmosphere+0xa8fdfd) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:
0:   Previous write of size 4 at 0x000005375c28 by main thread:
0:     #0 mpas_atmphys_driver_radiation_sw::radiation_sw_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:471:2 (mpas_atmosphere+0xc805f7) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#1 mpas_atmphys_driver_radiation_sw::driver_radiation_sw_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:877:7 (mpas_atmosphere+0xca19f1) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.102086.split2131 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:231:16 (mpas_atmosphere+0xa99c86) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0:     MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:229:7 (mpas_atmosphere+0xa8fdfd) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#5 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1036:15 (mpas_atmosphere+0xa10c41) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#6 atm_core::atm_core_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0xa0b767) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#7 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f92b3) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#8 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f3169) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#9 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f3160) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#10 main <null> (mpas_atmosphere+0x43aa4c) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:
0:   Location is global 'mpas_atmphys_vars::aer_opt_' of size 4 at 0x000005375c28 (mpas_atmosphere+0x5375c28)
0:
0:   Thread T2 'openmp_worker' (tid=3577284, running) created by main thread at:
0:     #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d4e) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5)
0:     MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x9fda57) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f8e34) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:
0: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:471:2 in mpas_atmphys_driver_radiation_sw::radiation_sw_from_mpas_
0: ==================

0: ==================
0: WARNING: ThreadSanitizer: data race (pid=3577273)
0:   Write of size 4 at 0x000005375c48 by thread T2:
0:     #0 mpas_atmphys_driver_pbl::pbl_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_pbl.F:392:8 (mpas_atmosphere+0xbcbf08) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#1 mpas_atmphys_driver_pbl::driver_pbl_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_pbl.F:818:7 (mpas_atmosphere+0xc06258) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.38.split2117 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:324:16 (mpas_atmosphere+0xa9601a) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0:     MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:322:7 (mpas_atmosphere+0xa92256) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:
0:   Previous write of size 4 at 0x000005375c48 by main thread:
0:     #0 mpas_atmphys_driver_pbl::pbl_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_pbl.F:392:8 (mpas_atmosphere+0xbcbf08) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#1 mpas_atmphys_driver_pbl::driver_pbl_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_pbl.F:818:7 (mpas_atmosphere+0xc06258) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.38.split2117 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:324:16 (mpas_atmosphere+0xa9601a) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0:     MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:322:7 (mpas_atmosphere+0xa92256) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#5 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1036:15 (mpas_atmosphere+0xa10c41) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#6 atm_core::atm_core_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0xa0b767) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#7 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f92b3) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#8 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f3169) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#9 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f3160) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#10 main <null> (mpas_atmosphere+0x43aa4c) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:
0:   Location is global 'mpas_atmphys_vars::ysu_pblmix_' of size 4 at 0x000005375c48 (mpas_atmosphere+0x5375c48)
0:
0:   Thread T2 'openmp_worker' (tid=3577284, running) created by main thread at:
0:     #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d4e) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5)
0:     MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x9fda57) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:     MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f8e34) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55)
0:
0: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_pbl.F:392:8 in mpas_atmphys_driver_pbl::pbl_from_mpas_
0: ==================
Two arrays (dzs_p and pin_p) are declared as one-dimensional arrays in mpas_atmphys_vars.
Both are alllocated in driver_lsm and driver_radiation_lw/sw allocate routines, which
are called outside parallel regions. They are global/shared arrays but access to their
elements from different threads does not depend on horizontal indexes, which means every
assignment must be done on a single thread only.

Here are thread sanitizer reports about potential data races

0: ==================
0: WARNING: ThreadSanitizer: data race (pid=448722)
0:   Write of size 4 at 0x7f64b1c0df88 by thread T2:
0:     #0 mpas_atmphys_driver_lsm::lsm_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_lsm.F:361:5 (mpas_atmosphere+0xb6cf8b) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#1 mpas_atmphys_driver_lsm::driver_lsm_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_lsm.F:697:7 (mpas_atmosphere+0xb9cf5e) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.262099.split2123 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:288:19 (mpas_atmosphere+0xa978bb) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0:     MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:286:7 (mpas_atmosphere+0xa91023) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:
0:   Previous write of size 4 at 0x7f64b1c0df88 by main thread:
0:     #0 mpas_atmphys_driver_lsm::lsm_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_lsm.F:361:5 (mpas_atmosphere+0xb6cf8b) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#1 mpas_atmphys_driver_lsm::driver_lsm_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_lsm.F:697:7 (mpas_atmosphere+0xb9cf5e) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.262099.split2123 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:288:19 (mpas_atmosphere+0xa978bb) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0:     MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:286:7 (mpas_atmosphere+0xa91023) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#5 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1036:15 (mpas_atmosphere+0xa10c41) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#6 atm_core::atm_core_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0xa0b767) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#7 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f92b3) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#8 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f3169) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#9 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f3160) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#10 main <null> (mpas_atmosphere+0x43aa4c) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:
0:   Thread T2 'openmp_worker' (tid=448734, running) created by main thread at:
0:     #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d4e) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5)
0:     MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x9fda57) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f8e34) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:
0: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_lsm.F:361:5 in mpas_atmphys_driver_lsm::lsm_from_mpas_
0: ==================

0: ==================
0: WARNING: ThreadSanitizer: data race (pid=447023)
0:   Write of size 4 at 0x7f310610d900 by thread T2:
0:     #0 mpas_atmphys_driver_radiation_sw::radiation_sw_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:602:14 (mpas_atmosphere+0xc8e4bf) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#1 mpas_atmphys_driver_radiation_sw::driver_radiation_sw_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:877:7 (mpas_atmosphere+0xca309b) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.102086.split2131 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:231:16 (mpas_atmosphere+0xa99c86) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0:     MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:229:7 (mpas_atmosphere+0xa8fdfd) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:
0:   Previous write of size 4 at 0x7f310610d900 by main thread:
0:     #0 mpas_atmphys_driver_radiation_sw::radiation_sw_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:602:14 (mpas_atmosphere+0xc8e4bf) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#1 mpas_atmphys_driver_radiation_sw::driver_radiation_sw_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:877:7 (mpas_atmosphere+0xca309b) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.102086.split2131 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:231:16 (mpas_atmosphere+0xa99c86) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0:     MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:229:7 (mpas_atmosphere+0xa8fdfd) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#5 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1036:15 (mpas_atmosphere+0xa10c41) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#6 atm_core::atm_core_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0xa0b767) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#7 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f92b3) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#8 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f3169) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#9 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f3160) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#10 main <null> (mpas_atmosphere+0x43aa4c) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:
0:   Thread T2 'openmp_worker' (tid=447034, running) created by main thread at:
0:     #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d4e) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5)
0:     MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x9fda57) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f8e34) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:
0: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:602:14 in mpas_atmphys_driver_radiation_sw::radiation_sw_from_mpas_
0: ==================

0: ==================
0: WARNING: ThreadSanitizer: data race (pid=447023)
0:   Write of size 4 at 0x7f310610d900 by thread T2:
0:     #0 mpas_atmphys_driver_radiation_lw::radiation_lw_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_lw.F:473:14 (mpas_atmosphere+0xc35420) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#1 mpas_atmphys_driver_radiation_lw::driver_radiation_lw_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_lw.F:842:7 (mpas_atmosphere+0xc52b75) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.142089.split2129 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:244:16 (mpas_atmosphere+0xa9925f) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0:     MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:242:7 (mpas_atmosphere+0xa900b2) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:
0:   Previous write of size 4 at 0x7f310610d900 by main thread:
0:     #0 mpas_atmphys_driver_radiation_lw::radiation_lw_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_lw.F:473:14 (mpas_atmosphere+0xc35420) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#1 mpas_atmphys_driver_radiation_lw::driver_radiation_lw_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_lw.F:842:7 (mpas_atmosphere+0xc52b75) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.142089.split2129 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:244:16 (mpas_atmosphere+0xa9925f) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0:     MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:242:7 (mpas_atmosphere+0xa900b2) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#5 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1036:15 (mpas_atmosphere+0xa10c41) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#6 atm_core::atm_core_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0xa0b767) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#7 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f92b3) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#8 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f3169) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#9 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f3160) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#10 main <null> (mpas_atmosphere+0x43aa4c) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:
0:   Thread T2 'openmp_worker' (tid=447034, running) created by main thread at:
0:     #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d4e) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5)
0:     MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x9fda57) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:     MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f8e34) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3)
0:
0: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_lw.F:473:14 in mpas_atmphys_driver_radiation_lw::radiation_lw_from_mpas_
0: ==================
…d to 'thread owned elements'

These data race warnings are because two horizontal do loops in atmphys_sfc_diagnostics go from 1 to nCellsSolve
These loops should actually iterate from its to ite. Threads must not write into horizontal array elements 'owned by other threads'

0: ==================
0: WARNING: ThreadSanitizer: data race (pid=448722)
0:   Write of size 4 at 0x7f649850fb84 by main thread:
0:     #0 mpas_atmphys_sfc_diagnostics::atmphys_sfc_diagnostics_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_sfc_diagnostics.F:85:14 (mpas_atmosphere+0x2686c1d) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#1 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.34.split2121 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:313:16 (mpas_atmosphere+0xa97098) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#2 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0:     MPAS-Dev#3 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:311:7 (mpas_atmosphere+0xa91edb) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#4 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1036:15 (mpas_atmosphere+0xa10c41) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#5 atm_core::atm_core_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0xa0b767) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#6 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f92b3) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#7 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f3169) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#8 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f3160) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#9 main <null> (mpas_atmosphere+0x43aa4c) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:
0:   Previous write of size 4 at 0x7f649850fb84 by thread T2:
0:     #0 mpas_atmphys_sfc_diagnostics::atmphys_sfc_diagnostics_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_sfc_diagnostics.F:85:14 (mpas_atmosphere+0x2686c1d) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#1 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.34.split2121 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:313:16 (mpas_atmosphere+0xa97098) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#2 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0:     MPAS-Dev#3 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:311:7 (mpas_atmosphere+0xa91edb) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:
0:   Thread T2 'openmp_worker' (tid=448734, running) created by main thread at:
0:     #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d4e) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5)
0:     MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x9fda57) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:     MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f8e34) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9)
0:
0: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_sfc_diagnostics.F:85:14 in mpas_atmphys_sfc_diagnostics::atmphys_sfc_diagnostics_
0: ==================
The shared array o3clim_p is allocated in allocate_radiation_lw with first dimension bounds 'ims:ime'.
In radiation_lw_from_MPAS which runs on multiple threads, values are assigned only to 'thraad owned'
array subsection 'its:ite'. When it's passed as an actual argument to vinterp_ozn, it is passed as

            call vinterp_ozn(1,ncols,ncols,nlevs,p2d,pin_p,num_oznlevels,o3clim_p(1,1,j),o32d)

and the corresponding dummy argument in vinterp_ozn is ozmix_in dimensioned as

  real(kind=RKIND), intent(in) :: ozmix_in(pcols,levsiz)

where pcols is equal to (ite-its+1).

Passing o3clim_p with a lower bound of 1 for the first dimension is incorrect on all but the first thread.
Actual argument must be an array subsection valid for a given thread, which is its:ite for the first dimension.

0: ==================
0: WARNING: ThreadSanitizer: data race (pid=381796)
0:   Read of size 4 at 0x7f5d70932064 by main thread:
0:     #0 module_ra_rrtmg_vinterp::vinterp_ozn_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/physics_wrf/module_ra_rrtmg_vinterp.F:92:10 (mpas_atmosphere+0x9ee6af) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:     MPAS-Dev#1 mpas_atmphys_driver_radiation_lw::radiation_lw_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_lw.F:496:19 (mpas_atmosphere+0x6dc097) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:     MPAS-Dev#2 mpas_atmphys_driver_radiation_lw::driver_radiation_lw_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_lw.F:849:7 (mpas_atmosphere+0x6e57ca) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:     MPAS-Dev#3 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.16.split1594 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:244:16 (mpas_atmosphere+0x64571c) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:     MPAS-Dev#4 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0:     MPAS-Dev#5 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:242:7 (mpas_atmosphere+0x64289d) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:     MPAS-Dev#6 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1036:15 (mpas_atmosphere+0x616631) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:     MPAS-Dev#7 atm_core::atm_core_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0x6142dc) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:     MPAS-Dev#8 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f5811) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:     MPAS-Dev#9 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f3195) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:     MPAS-Dev#10 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f318b) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:     MPAS-Dev#11 main <null> (mpas_atmosphere+0x43aa9c) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:
0:   Previous write of size 4 at 0x7f5d70932064 by thread T2:
0:     #0 mpas_atmphys_driver_radiation_lw::radiation_lw_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_lw.F:480:17 (mpas_atmosphere+0x6db879) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:     MPAS-Dev#1 mpas_atmphys_driver_radiation_lw::driver_radiation_lw_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_lw.F:849:7 (mpas_atmosphere+0x6e57ca) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:     MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.16.split1594 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:244:16 (mpas_atmosphere+0x64571c) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:     MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0:     MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:242:7 (mpas_atmosphere+0x64289d) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:
0:   Thread T2 'openmp_worker' (tid=381849, running) created by main thread at:
0:     #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d9e) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:     MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5)
0:     MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x60e765) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:     MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f511c) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:
0: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/physics_wrf/module_ra_rrtmg_vinterp.F:92:10 in module_ra_rrtmg_vinterp::vinterp_ozn_
0: ==================
…a single thread

This thread santitizer data race warning is about a shared array (rho_zz) being asssigened value
at horizontal index nCell+1 in a routine (atm_recover_large_step_variables_work) that runs on
all threads, so each thread is writing to the same memory location. Few other arrays that
also set the value at index nCell+1 are now restricted to a single thread

0: ==================
0: WARNING: ThreadSanitizer: data race (pid=3327569)
0:   Write of size 8 at 0x7f4fc3698458 by main thread:
0:     #0 atm_time_integration::atm_recover_large_step_variables_work_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:4390:10 (mpas_atmosphere+0xbd0728) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f)
0:     MPAS-Dev#1 atm_time_integration::atm_recover_large_step_variables_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:4294:12 (mpas_atmosphere+0xbcffe2) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f)
0:     MPAS-Dev#2 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.51.split10812 /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2427:21 (mpas_atmosphere+0xc1ad64) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f)
0:     MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0:     MPAS-Dev#4 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2425:7 (mpas_atmosphere+0xba9d47) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f)
0:     MPAS-Dev#5 atm_time_integration::atm_timestep_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:1829:15 (mpas_atmosphere+0xba377f) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f)
0:     MPAS-Dev#6 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1044:12 (mpas_atmosphere+0x616655) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f)
0:     MPAS-Dev#7 atm_core::atm_core_run_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0x6142dc) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f)
0:     MPAS-Dev#8 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f5811) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f)
0:     MPAS-Dev#9 MAIN /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f3195) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f)
0:     MPAS-Dev#10 MAIN /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f318b) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f)
0:     MPAS-Dev#11 main <null> (mpas_atmosphere+0x43aa9c) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f)
0:
0:   Previous write of size 8 at 0x7f4fc3698458 by thread T2:
0:     #0 atm_time_integration::atm_recover_large_step_variables_work_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:4390:10 (mpas_atmosphere+0xbd0728) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f)
0:     MPAS-Dev#1 atm_time_integration::atm_recover_large_step_variables_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:4294:12 (mpas_atmosphere+0xbcffe2) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f)
0:     MPAS-Dev#2 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.51.split10812 /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2427:21 (mpas_atmosphere+0xc1ad64) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f)
0:     MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0:     MPAS-Dev#4 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2425:7 (mpas_atmosphere+0xba9d47) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f)
0:
0:   Thread T2 'openmp_worker' (tid=3327621, running) created by main thread at:
0:     #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d9e) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f)
0:     MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5)
0:     MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x60e765) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f)
0:     MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f511c) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f)
0:
0: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:4390:10 in atm_time_integration::atm_recover_large_step_variables_work_
0: ==================
At two places in mpas_atm_time_integration.F, in atm_rk_integration_setup (theta_m_2)
and in atm_rk_dynamics_substep_finish (theta_m_1) zero is assigned to the cellEnd+1
element of those arrays on all threads. But cellEnd+1 is the same memory location
as cellStart on the following thread. To avoid data races only the last thread
should be allowed to write to cellEnd+1.

4: ==================
4: WARNING: ThreadSanitizer: data race (pid=3062683)
4:   Write of size 4 at 0x7f0f6ac5dbc0 by thread T3:
4:     #0 atm_time_integration::atm_rk_dynamics_substep_finish_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:7757:16 (mpas_atmosphere+0xbfe47f) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
4:     MPAS-Dev#1 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.71.split10818 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2645:18 (mpas_atmosphere+0xc1c942) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
4:     MPAS-Dev#2 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
4:     MPAS-Dev#3 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2643:7 (mpas_atmosphere+0xba65cd) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
4:
4:   Previous write of size 4 at 0x7f0f6ac5dbc0 by thread T2:
4:     #0 atm_time_integration::atm_rk_dynamics_substep_finish_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:7734:7 (mpas_atmosphere+0xbfcbf1) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
4:     MPAS-Dev#1 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.71.split10818 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2645:18 (mpas_atmosphere+0xc1c942) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
4:     MPAS-Dev#2 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
4:     MPAS-Dev#3 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2643:7 (mpas_atmosphere+0xba65cd) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
4:
4:   Thread T3 'openmp_worker' (tid=3062733, running) created by main thread at:
4:     #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d9e) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
4:     MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5)
4:     MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x60e765) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
4:     MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f511c) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
4:
4:   Thread T2 'openmp_worker' (tid=3062727, running) created by main thread at:
4:     #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d9e) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
4:     MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5)
4:     MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x60e765) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
4:     MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f511c) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
4:
4: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:7757:16 in atm_time_integration::atm_rk_dynamics_substep_finish_
4: ==================

1: ==================
1: WARNING: ThreadSanitizer: data race (pid=3062680)
1:   Write of size 4 at 0x7f5feac7ab28 by thread T7:
1:     #0 atm_time_integration::atm_rk_integration_setup_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:3158:13 (mpas_atmosphere+0xbb45bb) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
1:     MPAS-Dev#1 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.210748.split10795 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2113:15 (mpas_atmosphere+0xc17879) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
1:     MPAS-Dev#2 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
1:     MPAS-Dev#3 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2111:7 (mpas_atmosphere+0xba60c3) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
1:
1:   Previous write of size 4 at 0x7f5feac7ab28 by thread T6:
1:     #0 atm_time_integration::atm_rk_integration_setup_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:3139:7 (mpas_atmosphere+0xbb2c99) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
1:     MPAS-Dev#1 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.210748.split10795 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2113:15 (mpas_atmosphere+0xc17879) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
1:     MPAS-Dev#2 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
1:     MPAS-Dev#3 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2111:7 (mpas_atmosphere+0xba60c3) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
1:
1:   Thread T7 'openmp_worker' (tid=3062773, running) created by main thread at:
1:     #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d9e) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
1:     MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5)
1:     MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x60e765) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
1:     MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f511c) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
1:
1:   Thread T6 'openmp_worker' (tid=3062764, running) created by main thread at:
1:     #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d9e) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
1:     MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5)
1:     MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x60e765) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
1:     MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f511c) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3)
1:
1: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:3158:13 in atm_time_integration::atm_rk_integration_setup_
1: ==================
Vertical array cofrz is shared and only single thread should assign the values

1: ==================
1: WARNING: ThreadSanitizer: data race (pid=3066462)
1:   Write of size 4 at 0x7f29eed0de00 by thread T2:
1:     #0 atm_time_integration::atm_compute_vert_imp_coefs_work_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:3442:10 (mpas_atmosphere+0x21190fd) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc)
1:     MPAS-Dev#1 atm_time_integration::atm_compute_vert_imp_coefs_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:3368:12 (mpas_atmosphere+0x21184e6) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc)
1:     MPAS-Dev#2 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.1013079.split13105 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2191:18 (mpas_atmosphere+0x225cd69) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc)
1:     MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
1:     MPAS-Dev#4 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2189:7 (mpas_atmosphere+0x20ecc60) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc)
1:
1:   Previous write of size 4 at 0x7f29eed0de00 by main thread:
1:     #0 atm_time_integration::atm_compute_vert_imp_coefs_work_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:3442:10 (mpas_atmosphere+0x21190fd) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc)
1:     MPAS-Dev#1 atm_time_integration::atm_compute_vert_imp_coefs_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:3368:12 (mpas_atmosphere+0x21184e6) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc)
1:     MPAS-Dev#2 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.1013079.split13105 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2191:18 (mpas_atmosphere+0x225cd69) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc)
1:     MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
1:     MPAS-Dev#4 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2189:7 (mpas_atmosphere+0x20ecc60) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc)
1:     MPAS-Dev#5 atm_time_integration::atm_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:1833:15 (mpas_atmosphere+0x20e5e57) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc)
1:     MPAS-Dev#6 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1044:12 (mpas_atmosphere+0xa0e42b) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc)
1:     MPAS-Dev#7 atm_core::atm_core_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0xa09076) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc)
1:     MPAS-Dev#8 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f9170) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc)
1:     MPAS-Dev#9 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f316c) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc)
1:     MPAS-Dev#10 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f315d) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc)
1:     MPAS-Dev#11 main <null> (mpas_atmosphere+0x43aa4c) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc)
1:
1:   Thread T2 'openmp_worker' (tid=3066514, running) created by main thread at:
1:     #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d4e) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc)
1:     MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5)
1:     MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x9fb576) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc)
1:     MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f8d18) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc)
1:
1: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:3442:10 in atm_time_integration::atm_compute_vert_imp_coefs_work_
1: ==================
…_thompson

Two variables declared in MODULE module_mp_thompson are shared across all threads,
and are assigned values from different threads in mp_gt_driver.

Declare them as thread private.

There are many other variables (scalars or arrays) declared in module_mp_thompson
that are also shared, but thread sanitizer did not report potential data races.
This could be for several reasons. Either they are assigned values only in init
routine which is executed only on the main thread (fe. look up tables), or they
are multidimensional arrays that have one 'tile' dimension and different tile
elements are accessed on different threads, or maybe in my configuration that
I'm currently running a certain code path is never executed.

0: ==================
0: WARNING: ThreadSanitizer: data race (pid=3071161)
0:   Write of size 4 at 0x00000299bb78 by thread T2:
0:     #0 module::thompson::mp_gt_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/physics_wrf/module_mp_thompson.F:1145:10 (mpas_atmosphere+0x852a06) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01)
0:     MPAS-Dev#1 mpas_atmphys_driver_microphysics::driver_microphysics_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_microphysics.F:391:13 (mpas_atmosphere+0xd321cd) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01)
0:     MPAS-Dev#2 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.83.split10822 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2815:17 (mpas_atmosphere+0xc1d45c) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01)
0:     MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0:     MPAS-Dev#4 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2813:7 (mpas_atmosphere+0xbaf36a) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01)
0:
0:   Previous write of size 4 at 0x00000299bb78 by main thread:
0:     #0 module::thompson::mp_gt_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/physics_wrf/module_mp_thompson.F:1145:10 (mpas_atmosphere+0x852a06) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01)
0:     MPAS-Dev#1 mpas_atmphys_driver_microphysics::driver_microphysics_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_microphysics.F:391:13 (mpas_atmosphere+0xd321cd) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01)
0:     MPAS-Dev#2 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.83.split10822 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2815:17 (mpas_atmosphere+0xc1d45c) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01)
0:     MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0:     MPAS-Dev#4 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2813:7 (mpas_atmosphere+0xbaf36a) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01)
0:     MPAS-Dev#5 atm_time_integration::atm_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:1833:15 (mpas_atmosphere+0xba3b1f) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01)
0:     MPAS-Dev#6 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1044:12 (mpas_atmosphere+0x616655) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01)
0:     MPAS-Dev#7 atm_core::atm_core_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0x6142dc) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01)
0:     MPAS-Dev#8 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f5811) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01)
0:     MPAS-Dev#9 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f3195) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01)
0:     MPAS-Dev#10 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f318b) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01)
0:     MPAS-Dev#11 main <null> (mpas_atmosphere+0x43aa9c) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01)
0:
0:   Location is global 'module::thompson::nt_c_' of size 4 at 0x00000299bb78 (mpas_atmosphere+0x299bb78)
0:
0:   Thread T2 'openmp_worker' (tid=3071208, running) created by main thread at:
0:     #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d9e) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01)
0:     MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5)
0:     MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x60e765) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01)
0:     MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f511c) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01)
0:
0: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/physics_wrf/module_mp_thompson.F:1145:10 in module::thompson::mp_gt_driver_
0: ==================
This commit fixes a bug in 'sfclay_mynn routine' (in module_sf_mynn.F) where
3d arrays dimensioned with lower:upper bounds of the first dimension
(ims:ime, ...)  are passed as an actual arguments to 'sf_mynn_pre_run'
where dummy arguments are dimensioned with first dimension bounds its:ite.
Inside the 'sf_mynn_pre_run' routine the values are assigned to those dummy
arguments in loops: do  i = its,ite which means the first (ite-its+1) values,
corresponding to (ims:ims+ite-its) values of the actual argument are modified,
which is incorrect. In addition to being incorrect it is also producing
diffetent results between runs running on different number of threads.

There are three ways to fix this. Pass only the (its:ite, ...) subsection of
those 3d arrays as actual arguments from 'sfclay_mynn' matching the dummy
arguments' shape,or changing the lower:upper bounds of dummy arguments in
sf_mynn_pre_run to (ims:ime, ...), or maybe simply assign the values to the
resulting column arrays directly in 'sfclay_mynn' skipping the call to
'sf_mynn_pre_run' altogether.

This commit implements the first option and also provides an alternative
in #ifdef-ed code without calling 'sf_mynn_pre_run' at all.

I'm not sure which option is preferable. In my opinion, unless I'm missingi
some other reason, I do not see the need for a subroutine call here.
When floating point trap flags are enabled with the thread sanitizer
the following error stops the model execution:

0: forrtl: error (182): floating invalid - possible uninitialized real/complex variable.
0: Image              PC                Routine            Line        Source
0: mpas_atmosphere    000000000044B086  Unknown               Unknown  Unknown
0: mpas_atmosphere    000000000044B616  Unknown               Unknown  Unknown
0: libc.so.6          00007F886D086BF0  Unknown               Unknown  Unknown
0: mpas_atmosphere    000000000206F625  dotproduct                168  mpas_pv_diagnostics.F
0: mpas_atmosphere    000000000209D21D  calc_gradxu_cell         1032  mpas_pv_diagnostics.F
0: mpas_atmosphere    00000000020A6F72  calc_epv                 1232  mpas_pv_diagnostics.F
0: mpas_atmosphere    00000000020ABCB7  atm_compute_pv_di        1311  mpas_pv_diagnostics.F
0: mpas_atmosphere    000000000206EE6A  pv_diagnostics_co         144  mpas_pv_diagnostics.F
0: mpas_atmosphere    0000000001FB9A8A  mpas_atm_diag_com         116  mpas_atm_diagnostics_manager.F
0: mpas_atmosphere    0000000000A06BAB  atm_core_run              707  mpas_atm_core.F
0: mpas_atmosphere    00000000004F9171  mpas_run                  416  mpas_subdriver.F
0: mpas_atmosphere    00000000004F316D  mpas                       20  mpas.F
0: mpas_atmosphere    000000000043AA4D  Unknown               Unknown  Unknown
0: libc.so.6          00007F886D0715D0  Unknown               Unknown  Unknown
0: libc.so.6          00007F886D071680  __libc_start_main     Unknown  Unknown
0: mpas_atmosphere    000000000043A965  Unknown               Unknown  Unknown

This is due to a typo in calc_gradxu_cell. It seems this bug was causing the differences
between threaded and non-threaded runs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant