Fix OpenMP reproducibility#1480
Open
DusanJovic-NOAA wants to merge 11 commits into
Open
Conversation
ThreadSanitizer warns about a potential data race. Two variables (aer_opt and ysu_pblmix) are defined in mpas_atmphys_vars module and are global (a single copy shared by all threads). In this specific case all threads assign 0 to both variables, which means even if there's a data race the result will be the same regardless of the thread count. But I think this should be fixed if nothing else then to have clean(er) sanitizer report, which will make it easier to spot the actual data races later on. 0: ================== 0: WARNING: ThreadSanitizer: data race (pid=3577273) 0: Write of size 4 at 0x000005375c28 by thread T2: 0: #0 mpas_atmphys_driver_radiation_sw::radiation_sw_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:471:2 (mpas_atmosphere+0xc805f7) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#1 mpas_atmphys_driver_radiation_sw::driver_radiation_sw_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:877:7 (mpas_atmosphere+0xca19f1) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.102086.split2131 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:231:16 (mpas_atmosphere+0xa99c86) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 0: MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:229:7 (mpas_atmosphere+0xa8fdfd) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: 0: Previous write of size 4 at 0x000005375c28 by main thread: 0: #0 mpas_atmphys_driver_radiation_sw::radiation_sw_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:471:2 (mpas_atmosphere+0xc805f7) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#1 mpas_atmphys_driver_radiation_sw::driver_radiation_sw_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:877:7 (mpas_atmosphere+0xca19f1) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.102086.split2131 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:231:16 (mpas_atmosphere+0xa99c86) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 0: MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:229:7 (mpas_atmosphere+0xa8fdfd) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#5 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1036:15 (mpas_atmosphere+0xa10c41) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#6 atm_core::atm_core_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0xa0b767) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#7 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f92b3) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#8 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f3169) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#9 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f3160) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#10 main <null> (mpas_atmosphere+0x43aa4c) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: 0: Location is global 'mpas_atmphys_vars::aer_opt_' of size 4 at 0x000005375c28 (mpas_atmosphere+0x5375c28) 0: 0: Thread T2 'openmp_worker' (tid=3577284, running) created by main thread at: 0: #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d4e) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5) 0: MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x9fda57) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f8e34) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: 0: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:471:2 in mpas_atmphys_driver_radiation_sw::radiation_sw_from_mpas_ 0: ================== 0: ================== 0: WARNING: ThreadSanitizer: data race (pid=3577273) 0: Write of size 4 at 0x000005375c48 by thread T2: 0: #0 mpas_atmphys_driver_pbl::pbl_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_pbl.F:392:8 (mpas_atmosphere+0xbcbf08) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#1 mpas_atmphys_driver_pbl::driver_pbl_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_pbl.F:818:7 (mpas_atmosphere+0xc06258) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.38.split2117 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:324:16 (mpas_atmosphere+0xa9601a) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 0: MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:322:7 (mpas_atmosphere+0xa92256) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: 0: Previous write of size 4 at 0x000005375c48 by main thread: 0: #0 mpas_atmphys_driver_pbl::pbl_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_pbl.F:392:8 (mpas_atmosphere+0xbcbf08) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#1 mpas_atmphys_driver_pbl::driver_pbl_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_pbl.F:818:7 (mpas_atmosphere+0xc06258) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.38.split2117 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:324:16 (mpas_atmosphere+0xa9601a) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 0: MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:322:7 (mpas_atmosphere+0xa92256) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#5 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1036:15 (mpas_atmosphere+0xa10c41) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#6 atm_core::atm_core_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0xa0b767) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#7 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f92b3) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#8 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f3169) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#9 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f3160) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#10 main <null> (mpas_atmosphere+0x43aa4c) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: 0: Location is global 'mpas_atmphys_vars::ysu_pblmix_' of size 4 at 0x000005375c48 (mpas_atmosphere+0x5375c48) 0: 0: Thread T2 'openmp_worker' (tid=3577284, running) created by main thread at: 0: #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d4e) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5) 0: MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x9fda57) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f8e34) (BuildId: 1db7011d81c236acf1d054158b592f58bdcdcc55) 0: 0: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_pbl.F:392:8 in mpas_atmphys_driver_pbl::pbl_from_mpas_ 0: ==================
Two arrays (dzs_p and pin_p) are declared as one-dimensional arrays in mpas_atmphys_vars. Both are alllocated in driver_lsm and driver_radiation_lw/sw allocate routines, which are called outside parallel regions. They are global/shared arrays but access to their elements from different threads does not depend on horizontal indexes, which means every assignment must be done on a single thread only. Here are thread sanitizer reports about potential data races 0: ================== 0: WARNING: ThreadSanitizer: data race (pid=448722) 0: Write of size 4 at 0x7f64b1c0df88 by thread T2: 0: #0 mpas_atmphys_driver_lsm::lsm_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_lsm.F:361:5 (mpas_atmosphere+0xb6cf8b) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#1 mpas_atmphys_driver_lsm::driver_lsm_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_lsm.F:697:7 (mpas_atmosphere+0xb9cf5e) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.262099.split2123 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:288:19 (mpas_atmosphere+0xa978bb) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 0: MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:286:7 (mpas_atmosphere+0xa91023) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: 0: Previous write of size 4 at 0x7f64b1c0df88 by main thread: 0: #0 mpas_atmphys_driver_lsm::lsm_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_lsm.F:361:5 (mpas_atmosphere+0xb6cf8b) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#1 mpas_atmphys_driver_lsm::driver_lsm_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_lsm.F:697:7 (mpas_atmosphere+0xb9cf5e) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.262099.split2123 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:288:19 (mpas_atmosphere+0xa978bb) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 0: MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:286:7 (mpas_atmosphere+0xa91023) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#5 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1036:15 (mpas_atmosphere+0xa10c41) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#6 atm_core::atm_core_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0xa0b767) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#7 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f92b3) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#8 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f3169) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#9 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f3160) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#10 main <null> (mpas_atmosphere+0x43aa4c) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: 0: Thread T2 'openmp_worker' (tid=448734, running) created by main thread at: 0: #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d4e) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5) 0: MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x9fda57) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f8e34) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: 0: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_lsm.F:361:5 in mpas_atmphys_driver_lsm::lsm_from_mpas_ 0: ================== 0: ================== 0: WARNING: ThreadSanitizer: data race (pid=447023) 0: Write of size 4 at 0x7f310610d900 by thread T2: 0: #0 mpas_atmphys_driver_radiation_sw::radiation_sw_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:602:14 (mpas_atmosphere+0xc8e4bf) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#1 mpas_atmphys_driver_radiation_sw::driver_radiation_sw_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:877:7 (mpas_atmosphere+0xca309b) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.102086.split2131 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:231:16 (mpas_atmosphere+0xa99c86) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 0: MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:229:7 (mpas_atmosphere+0xa8fdfd) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: 0: Previous write of size 4 at 0x7f310610d900 by main thread: 0: #0 mpas_atmphys_driver_radiation_sw::radiation_sw_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:602:14 (mpas_atmosphere+0xc8e4bf) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#1 mpas_atmphys_driver_radiation_sw::driver_radiation_sw_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:877:7 (mpas_atmosphere+0xca309b) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.102086.split2131 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:231:16 (mpas_atmosphere+0xa99c86) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 0: MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:229:7 (mpas_atmosphere+0xa8fdfd) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#5 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1036:15 (mpas_atmosphere+0xa10c41) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#6 atm_core::atm_core_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0xa0b767) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#7 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f92b3) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#8 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f3169) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#9 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f3160) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#10 main <null> (mpas_atmosphere+0x43aa4c) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: 0: Thread T2 'openmp_worker' (tid=447034, running) created by main thread at: 0: #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d4e) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5) 0: MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x9fda57) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f8e34) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: 0: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_sw.F:602:14 in mpas_atmphys_driver_radiation_sw::radiation_sw_from_mpas_ 0: ================== 0: ================== 0: WARNING: ThreadSanitizer: data race (pid=447023) 0: Write of size 4 at 0x7f310610d900 by thread T2: 0: #0 mpas_atmphys_driver_radiation_lw::radiation_lw_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_lw.F:473:14 (mpas_atmosphere+0xc35420) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#1 mpas_atmphys_driver_radiation_lw::driver_radiation_lw_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_lw.F:842:7 (mpas_atmosphere+0xc52b75) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.142089.split2129 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:244:16 (mpas_atmosphere+0xa9925f) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 0: MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:242:7 (mpas_atmosphere+0xa900b2) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: 0: Previous write of size 4 at 0x7f310610d900 by main thread: 0: #0 mpas_atmphys_driver_radiation_lw::radiation_lw_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_lw.F:473:14 (mpas_atmosphere+0xc35420) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#1 mpas_atmphys_driver_radiation_lw::driver_radiation_lw_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_lw.F:842:7 (mpas_atmosphere+0xc52b75) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.142089.split2129 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:244:16 (mpas_atmosphere+0xa9925f) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 0: MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:242:7 (mpas_atmosphere+0xa900b2) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#5 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1036:15 (mpas_atmosphere+0xa10c41) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#6 atm_core::atm_core_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0xa0b767) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#7 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f92b3) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#8 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f3169) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#9 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f3160) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#10 main <null> (mpas_atmosphere+0x43aa4c) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: 0: Thread T2 'openmp_worker' (tid=447034, running) created by main thread at: 0: #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d4e) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5) 0: MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x9fda57) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f8e34) (BuildId: ebe0b2300b84755b756c0b6ea7e42da9068a3aa3) 0: 0: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_lw.F:473:14 in mpas_atmphys_driver_radiation_lw::radiation_lw_from_mpas_ 0: ==================
…d to 'thread owned elements' These data race warnings are because two horizontal do loops in atmphys_sfc_diagnostics go from 1 to nCellsSolve These loops should actually iterate from its to ite. Threads must not write into horizontal array elements 'owned by other threads' 0: ================== 0: WARNING: ThreadSanitizer: data race (pid=448722) 0: Write of size 4 at 0x7f649850fb84 by main thread: 0: #0 mpas_atmphys_sfc_diagnostics::atmphys_sfc_diagnostics_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_sfc_diagnostics.F:85:14 (mpas_atmosphere+0x2686c1d) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#1 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.34.split2121 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:313:16 (mpas_atmosphere+0xa97098) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#2 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 0: MPAS-Dev#3 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:311:7 (mpas_atmosphere+0xa91edb) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#4 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1036:15 (mpas_atmosphere+0xa10c41) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#5 atm_core::atm_core_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0xa0b767) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#6 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f92b3) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#7 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f3169) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#8 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f3160) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#9 main <null> (mpas_atmosphere+0x43aa4c) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: 0: Previous write of size 4 at 0x7f649850fb84 by thread T2: 0: #0 mpas_atmphys_sfc_diagnostics::atmphys_sfc_diagnostics_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_sfc_diagnostics.F:85:14 (mpas_atmosphere+0x2686c1d) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#1 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.34.split2121 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:313:16 (mpas_atmosphere+0xa97098) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#2 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 0: MPAS-Dev#3 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:311:7 (mpas_atmosphere+0xa91edb) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: 0: Thread T2 'openmp_worker' (tid=448734, running) created by main thread at: 0: #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d4e) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5) 0: MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x9fda57) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f8e34) (BuildId: 6accb5e96f6fa6dfe5760f5ea3bf82167d3b3be9) 0: 0: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_sfc_diagnostics.F:85:14 in mpas_atmphys_sfc_diagnostics::atmphys_sfc_diagnostics_ 0: ==================
The shared array o3clim_p is allocated in allocate_radiation_lw with first dimension bounds 'ims:ime'.
In radiation_lw_from_MPAS which runs on multiple threads, values are assigned only to 'thraad owned'
array subsection 'its:ite'. When it's passed as an actual argument to vinterp_ozn, it is passed as
call vinterp_ozn(1,ncols,ncols,nlevs,p2d,pin_p,num_oznlevels,o3clim_p(1,1,j),o32d)
and the corresponding dummy argument in vinterp_ozn is ozmix_in dimensioned as
real(kind=RKIND), intent(in) :: ozmix_in(pcols,levsiz)
where pcols is equal to (ite-its+1).
Passing o3clim_p with a lower bound of 1 for the first dimension is incorrect on all but the first thread.
Actual argument must be an array subsection valid for a given thread, which is its:ite for the first dimension.
0: ==================
0: WARNING: ThreadSanitizer: data race (pid=381796)
0: Read of size 4 at 0x7f5d70932064 by main thread:
0: #0 module_ra_rrtmg_vinterp::vinterp_ozn_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/physics_wrf/module_ra_rrtmg_vinterp.F:92:10 (mpas_atmosphere+0x9ee6af) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0: MPAS-Dev#1 mpas_atmphys_driver_radiation_lw::radiation_lw_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_lw.F:496:19 (mpas_atmosphere+0x6dc097) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0: MPAS-Dev#2 mpas_atmphys_driver_radiation_lw::driver_radiation_lw_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_lw.F:849:7 (mpas_atmosphere+0x6e57ca) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0: MPAS-Dev#3 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.16.split1594 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:244:16 (mpas_atmosphere+0x64571c) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0: MPAS-Dev#4 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0: MPAS-Dev#5 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:242:7 (mpas_atmosphere+0x64289d) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0: MPAS-Dev#6 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1036:15 (mpas_atmosphere+0x616631) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0: MPAS-Dev#7 atm_core::atm_core_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0x6142dc) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0: MPAS-Dev#8 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f5811) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0: MPAS-Dev#9 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f3195) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0: MPAS-Dev#10 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f318b) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0: MPAS-Dev#11 main <null> (mpas_atmosphere+0x43aa9c) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:
0: Previous write of size 4 at 0x7f5d70932064 by thread T2:
0: #0 mpas_atmphys_driver_radiation_lw::radiation_lw_from_mpas_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_lw.F:480:17 (mpas_atmosphere+0x6db879) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0: MPAS-Dev#1 mpas_atmphys_driver_radiation_lw::driver_radiation_lw_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_radiation_lw.F:849:7 (mpas_atmosphere+0x6e57ca) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0: MPAS-Dev#2 mpas_atmphys_driver::physics_driver_.DIR.OMP.PARALLEL.LOOP.16.split1594 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:244:16 (mpas_atmosphere+0x64571c) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0: MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8)
0: MPAS-Dev#4 mpas_atmphys_driver::physics_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver.F:242:7 (mpas_atmosphere+0x64289d) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:
0: Thread T2 'openmp_worker' (tid=381849, running) created by main thread at:
0: #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d9e) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0: MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5)
0: MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x60e765) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0: MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f511c) (BuildId: 661a0696f86a62568e541fd06a7b066a026681b3)
0:
0: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/physics_wrf/module_ra_rrtmg_vinterp.F:92:10 in module_ra_rrtmg_vinterp::vinterp_ozn_
0: ==================
…a single thread This thread santitizer data race warning is about a shared array (rho_zz) being asssigened value at horizontal index nCell+1 in a routine (atm_recover_large_step_variables_work) that runs on all threads, so each thread is writing to the same memory location. Few other arrays that also set the value at index nCell+1 are now restricted to a single thread 0: ================== 0: WARNING: ThreadSanitizer: data race (pid=3327569) 0: Write of size 8 at 0x7f4fc3698458 by main thread: 0: #0 atm_time_integration::atm_recover_large_step_variables_work_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:4390:10 (mpas_atmosphere+0xbd0728) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f) 0: MPAS-Dev#1 atm_time_integration::atm_recover_large_step_variables_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:4294:12 (mpas_atmosphere+0xbcffe2) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f) 0: MPAS-Dev#2 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.51.split10812 /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2427:21 (mpas_atmosphere+0xc1ad64) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f) 0: MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 0: MPAS-Dev#4 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2425:7 (mpas_atmosphere+0xba9d47) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f) 0: MPAS-Dev#5 atm_time_integration::atm_timestep_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:1829:15 (mpas_atmosphere+0xba377f) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f) 0: MPAS-Dev#6 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1044:12 (mpas_atmosphere+0x616655) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f) 0: MPAS-Dev#7 atm_core::atm_core_run_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0x6142dc) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f) 0: MPAS-Dev#8 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f5811) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f) 0: MPAS-Dev#9 MAIN /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f3195) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f) 0: MPAS-Dev#10 MAIN /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f318b) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f) 0: MPAS-Dev#11 main <null> (mpas_atmosphere+0x43aa9c) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f) 0: 0: Previous write of size 8 at 0x7f4fc3698458 by thread T2: 0: #0 atm_time_integration::atm_recover_large_step_variables_work_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:4390:10 (mpas_atmosphere+0xbd0728) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f) 0: MPAS-Dev#1 atm_time_integration::atm_recover_large_step_variables_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:4294:12 (mpas_atmosphere+0xbcffe2) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f) 0: MPAS-Dev#2 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.51.split10812 /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2427:21 (mpas_atmosphere+0xc1ad64) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f) 0: MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 0: MPAS-Dev#4 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2425:7 (mpas_atmosphere+0xba9d47) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f) 0: 0: Thread T2 'openmp_worker' (tid=3327621, running) created by main thread at: 0: #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d9e) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f) 0: MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5) 0: MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x60e765) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f) 0: MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f511c) (BuildId: 1106bfe5a75efd6bf52d3fef2ee2ba7468c1b91f) 0: 0: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/dev/mpas/simple-mpas/src/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:4390:10 in atm_time_integration::atm_recover_large_step_variables_work_ 0: ==================
At two places in mpas_atm_time_integration.F, in atm_rk_integration_setup (theta_m_2) and in atm_rk_dynamics_substep_finish (theta_m_1) zero is assigned to the cellEnd+1 element of those arrays on all threads. But cellEnd+1 is the same memory location as cellStart on the following thread. To avoid data races only the last thread should be allowed to write to cellEnd+1. 4: ================== 4: WARNING: ThreadSanitizer: data race (pid=3062683) 4: Write of size 4 at 0x7f0f6ac5dbc0 by thread T3: 4: #0 atm_time_integration::atm_rk_dynamics_substep_finish_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:7757:16 (mpas_atmosphere+0xbfe47f) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 4: MPAS-Dev#1 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.71.split10818 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2645:18 (mpas_atmosphere+0xc1c942) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 4: MPAS-Dev#2 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 4: MPAS-Dev#3 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2643:7 (mpas_atmosphere+0xba65cd) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 4: 4: Previous write of size 4 at 0x7f0f6ac5dbc0 by thread T2: 4: #0 atm_time_integration::atm_rk_dynamics_substep_finish_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:7734:7 (mpas_atmosphere+0xbfcbf1) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 4: MPAS-Dev#1 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.71.split10818 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2645:18 (mpas_atmosphere+0xc1c942) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 4: MPAS-Dev#2 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 4: MPAS-Dev#3 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2643:7 (mpas_atmosphere+0xba65cd) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 4: 4: Thread T3 'openmp_worker' (tid=3062733, running) created by main thread at: 4: #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d9e) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 4: MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5) 4: MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x60e765) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 4: MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f511c) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 4: 4: Thread T2 'openmp_worker' (tid=3062727, running) created by main thread at: 4: #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d9e) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 4: MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5) 4: MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x60e765) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 4: MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f511c) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 4: 4: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:7757:16 in atm_time_integration::atm_rk_dynamics_substep_finish_ 4: ================== 1: ================== 1: WARNING: ThreadSanitizer: data race (pid=3062680) 1: Write of size 4 at 0x7f5feac7ab28 by thread T7: 1: #0 atm_time_integration::atm_rk_integration_setup_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:3158:13 (mpas_atmosphere+0xbb45bb) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 1: MPAS-Dev#1 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.210748.split10795 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2113:15 (mpas_atmosphere+0xc17879) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 1: MPAS-Dev#2 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 1: MPAS-Dev#3 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2111:7 (mpas_atmosphere+0xba60c3) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 1: 1: Previous write of size 4 at 0x7f5feac7ab28 by thread T6: 1: #0 atm_time_integration::atm_rk_integration_setup_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:3139:7 (mpas_atmosphere+0xbb2c99) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 1: MPAS-Dev#1 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.210748.split10795 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2113:15 (mpas_atmosphere+0xc17879) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 1: MPAS-Dev#2 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 1: MPAS-Dev#3 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2111:7 (mpas_atmosphere+0xba60c3) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 1: 1: Thread T7 'openmp_worker' (tid=3062773, running) created by main thread at: 1: #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d9e) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 1: MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5) 1: MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x60e765) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 1: MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f511c) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 1: 1: Thread T6 'openmp_worker' (tid=3062764, running) created by main thread at: 1: #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d9e) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 1: MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5) 1: MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x60e765) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 1: MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f511c) (BuildId: 1beca6f25de1e63cd10f0c4f537870fb672cadc3) 1: 1: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:3158:13 in atm_time_integration::atm_rk_integration_setup_ 1: ==================
Vertical array cofrz is shared and only single thread should assign the values 1: ================== 1: WARNING: ThreadSanitizer: data race (pid=3066462) 1: Write of size 4 at 0x7f29eed0de00 by thread T2: 1: #0 atm_time_integration::atm_compute_vert_imp_coefs_work_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:3442:10 (mpas_atmosphere+0x21190fd) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc) 1: MPAS-Dev#1 atm_time_integration::atm_compute_vert_imp_coefs_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:3368:12 (mpas_atmosphere+0x21184e6) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc) 1: MPAS-Dev#2 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.1013079.split13105 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2191:18 (mpas_atmosphere+0x225cd69) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc) 1: MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 1: MPAS-Dev#4 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2189:7 (mpas_atmosphere+0x20ecc60) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc) 1: 1: Previous write of size 4 at 0x7f29eed0de00 by main thread: 1: #0 atm_time_integration::atm_compute_vert_imp_coefs_work_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:3442:10 (mpas_atmosphere+0x21190fd) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc) 1: MPAS-Dev#1 atm_time_integration::atm_compute_vert_imp_coefs_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:3368:12 (mpas_atmosphere+0x21184e6) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc) 1: MPAS-Dev#2 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.1013079.split13105 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2191:18 (mpas_atmosphere+0x225cd69) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc) 1: MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 1: MPAS-Dev#4 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2189:7 (mpas_atmosphere+0x20ecc60) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc) 1: MPAS-Dev#5 atm_time_integration::atm_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:1833:15 (mpas_atmosphere+0x20e5e57) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc) 1: MPAS-Dev#6 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1044:12 (mpas_atmosphere+0xa0e42b) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc) 1: MPAS-Dev#7 atm_core::atm_core_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0xa09076) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc) 1: MPAS-Dev#8 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f9170) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc) 1: MPAS-Dev#9 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f316c) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc) 1: MPAS-Dev#10 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f315d) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc) 1: MPAS-Dev#11 main <null> (mpas_atmosphere+0x43aa4c) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc) 1: 1: Thread T2 'openmp_worker' (tid=3066514, running) created by main thread at: 1: #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d4e) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc) 1: MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5) 1: MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x9fb576) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc) 1: MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f8d18) (BuildId: f8ba73f4bd7c32ffd1a891f5c1fd8932ce0ca4cc) 1: 1: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:3442:10 in atm_time_integration::atm_compute_vert_imp_coefs_work_ 1: ==================
…_thompson Two variables declared in MODULE module_mp_thompson are shared across all threads, and are assigned values from different threads in mp_gt_driver. Declare them as thread private. There are many other variables (scalars or arrays) declared in module_mp_thompson that are also shared, but thread sanitizer did not report potential data races. This could be for several reasons. Either they are assigned values only in init routine which is executed only on the main thread (fe. look up tables), or they are multidimensional arrays that have one 'tile' dimension and different tile elements are accessed on different threads, or maybe in my configuration that I'm currently running a certain code path is never executed. 0: ================== 0: WARNING: ThreadSanitizer: data race (pid=3071161) 0: Write of size 4 at 0x00000299bb78 by thread T2: 0: #0 module::thompson::mp_gt_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/physics_wrf/module_mp_thompson.F:1145:10 (mpas_atmosphere+0x852a06) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01) 0: MPAS-Dev#1 mpas_atmphys_driver_microphysics::driver_microphysics_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_microphysics.F:391:13 (mpas_atmosphere+0xd321cd) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01) 0: MPAS-Dev#2 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.83.split10822 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2815:17 (mpas_atmosphere+0xc1d45c) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01) 0: MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 0: MPAS-Dev#4 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2813:7 (mpas_atmosphere+0xbaf36a) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01) 0: 0: Previous write of size 4 at 0x00000299bb78 by main thread: 0: #0 module::thompson::mp_gt_driver_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/physics_wrf/module_mp_thompson.F:1145:10 (mpas_atmosphere+0x852a06) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01) 0: MPAS-Dev#1 mpas_atmphys_driver_microphysics::driver_microphysics_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/mpas_atmphys_driver_microphysics.F:391:13 (mpas_atmosphere+0xd321cd) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01) 0: MPAS-Dev#2 atm_time_integration::atm_srk3_.DIR.OMP.PARALLEL.LOOP.83.split10822 /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2815:17 (mpas_atmosphere+0xc1d45c) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01) 0: MPAS-Dev#3 __kmp_invoke_microtask <null> (libiomp5.so+0x14fdc8) 0: MPAS-Dev#4 atm_time_integration::atm_srk3_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:2813:7 (mpas_atmosphere+0xbaf36a) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01) 0: MPAS-Dev#5 atm_time_integration::atm_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/dynamics/mpas_atm_time_integration.F:1833:15 (mpas_atmosphere+0xba3b1f) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01) 0: MPAS-Dev#6 atm_core::atm_do_timestep_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:1044:12 (mpas_atmosphere+0x616655) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01) 0: MPAS-Dev#7 atm_core::atm_core_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:816:15 (mpas_atmosphere+0x6142dc) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01) 0: MPAS-Dev#8 mpas_subdriver::mpas_run_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:416:14 (mpas_atmosphere+0x4f5811) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01) 0: MPAS-Dev#9 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:20:9 (mpas_atmosphere+0x4f3195) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01) 0: MPAS-Dev#10 MAIN /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas.F:18:9 (mpas_atmosphere+0x4f318b) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01) 0: MPAS-Dev#11 main <null> (mpas_atmosphere+0x43aa9c) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01) 0: 0: Location is global 'module::thompson::nt_c_' of size 4 at 0x00000299bb78 (mpas_atmosphere+0x299bb78) 0: 0: Thread T2 'openmp_worker' (tid=3071208, running) created by main thread at: 0: #0 pthread_create /netbatch/donb58527_00/runDir/dir/workspace/NIT/xmain-rel/LX/xmainefi2linux_release/ws/icsws/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1090:3 (mpas_atmosphere+0x442d9e) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01) 0: MPAS-Dev#1 __kmp_create_worker /localdisk/promo-build/20250910/tmp/lin_32e-rtl_int_5_nor_dyn.rel.c0.s0.tcm1.t1..h1.u1-anompclxlin01/../../src/z_Linux_util.cpp:1015:7 (libiomp5.so+0x1508e5) 0: MPAS-Dev#2 atm_core::atm_core_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/mpas_atm_core.F:78:12 (mpas_atmosphere+0x60e765) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01) 0: MPAS-Dev#3 mpas_subdriver::mpas_init_ /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/driver/mpas_subdriver.F:394:14 (mpas_atmosphere+0x4f511c) (BuildId: 99aefae946b3e065f9008f4ae5a97210f82d7b01) 0: 0: SUMMARY: ThreadSanitizer: data race /scratch3/NCEPDEV/stmp/Dusan.Jovic/MPAS-Model-ncar-dev/src/core_atmosphere/physics/physics_wrf/module_mp_thompson.F:1145:10 in module::thompson::mp_gt_driver_ 0: ==================
This commit fixes a bug in 'sfclay_mynn routine' (in module_sf_mynn.F) where 3d arrays dimensioned with lower:upper bounds of the first dimension (ims:ime, ...) are passed as an actual arguments to 'sf_mynn_pre_run' where dummy arguments are dimensioned with first dimension bounds its:ite. Inside the 'sf_mynn_pre_run' routine the values are assigned to those dummy arguments in loops: do i = its,ite which means the first (ite-its+1) values, corresponding to (ims:ims+ite-its) values of the actual argument are modified, which is incorrect. In addition to being incorrect it is also producing diffetent results between runs running on different number of threads. There are three ways to fix this. Pass only the (its:ite, ...) subsection of those 3d arrays as actual arguments from 'sfclay_mynn' matching the dummy arguments' shape,or changing the lower:upper bounds of dummy arguments in sf_mynn_pre_run to (ims:ime, ...), or maybe simply assign the values to the resulting column arrays directly in 'sfclay_mynn' skipping the call to 'sf_mynn_pre_run' altogether. This commit implements the first option and also provides an alternative in #ifdef-ed code without calling 'sf_mynn_pre_run' at all. I'm not sure which option is preferable. In my opinion, unless I'm missingi some other reason, I do not see the need for a subroutine call here.
When floating point trap flags are enabled with the thread sanitizer the following error stops the model execution: 0: forrtl: error (182): floating invalid - possible uninitialized real/complex variable. 0: Image PC Routine Line Source 0: mpas_atmosphere 000000000044B086 Unknown Unknown Unknown 0: mpas_atmosphere 000000000044B616 Unknown Unknown Unknown 0: libc.so.6 00007F886D086BF0 Unknown Unknown Unknown 0: mpas_atmosphere 000000000206F625 dotproduct 168 mpas_pv_diagnostics.F 0: mpas_atmosphere 000000000209D21D calc_gradxu_cell 1032 mpas_pv_diagnostics.F 0: mpas_atmosphere 00000000020A6F72 calc_epv 1232 mpas_pv_diagnostics.F 0: mpas_atmosphere 00000000020ABCB7 atm_compute_pv_di 1311 mpas_pv_diagnostics.F 0: mpas_atmosphere 000000000206EE6A pv_diagnostics_co 144 mpas_pv_diagnostics.F 0: mpas_atmosphere 0000000001FB9A8A mpas_atm_diag_com 116 mpas_atm_diagnostics_manager.F 0: mpas_atmosphere 0000000000A06BAB atm_core_run 707 mpas_atm_core.F 0: mpas_atmosphere 00000000004F9171 mpas_run 416 mpas_subdriver.F 0: mpas_atmosphere 00000000004F316D mpas 20 mpas.F 0: mpas_atmosphere 000000000043AA4D Unknown Unknown Unknown 0: libc.so.6 00007F886D0715D0 Unknown Unknown Unknown 0: libc.so.6 00007F886D071680 __libc_start_main Unknown Unknown 0: mpas_atmosphere 000000000043A965 Unknown Unknown Unknown This is due to a typo in calc_gradxu_cell. It seems this bug was causing the differences between threaded and non-threaded runs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix the reproducibility of model outputs when a threaded build runs on a different number of OpenMP threads.
I used thread sanitizer to find potential data races, which frequently cause model non-reproducibility. Not every data race results in non-reproducible output, for example, a data race where multiple threads write the same initial value to a shared memory location. I think those cases should be fixed as well to minimize the number of warnings the thread sanitizer reports.
I tested these code changes using Intel (2025.3.0) and GNU (11.5.0). To get reproducible results with optimized code (
-O3) for the Intel compiler I had to use the-fp-model precisecompiler flag, in addition to the flags currently specified in the CMake files. For the GNU compiler I had to use-fno-tree-loop-vectorize. I didn't add those flags to the CMake files because I haven't checked the performance impact.With the code changes included in this PR and the compiler flags mentioned above, I am getting reproducible outputs (history, diag and restart files, checked by running nccmp) when I use 'mesoscale_reference' physics. For 'convection_permitting' physics two additional changes are needed in physics_mmm:
and