How Do HPC Programmers Mix MPC and SMC in Distributed Computing Environment?

In today's distributed computing environment where clustered symmetrical multi-processing (SMP) nodes interconnect via high-speed network, message passing concurrency (MPC) model is perfectly applicable to inter-node communication and shared memory concurrency (SMC) model applies to intra-node computation. Mixing MPC with SMC in distributed applications is a common programming practice. However, it's unclear how high-performance computing (HPC) developers mix concurrency models. We investigated twenty-two representative mixing HPC applications collected from super-computing laboratories, research-purpose benchmarks and the open source world. Our investigation shows that there are mainly two mixing modes, namely separated mode and tangled mode. Among twenty-two applications, fifteen mix in separated mode while seven others perform mixing in tangled mode. We further find tangled mode is more complicated and error prone than separated mode. We therefore suggest when performing mixing, distributed applications should mix MPC with SMC in separated mode if possible.