The Egyptian National HPC Grid (EN-HPCG): Open-Source Slurm Implementation from Cluster to Grid Approach

Abstract Recently, Egypt has recognized the pivotal role of High-Performance Computing in advancing science and innovation. The Egyptian National High Performance Computing Grid (EN-HPCG) advocates for consolidating high-performance computing resources, creating a unified pool accessible to all universities and scientific research centers using the PBS Pro scheduler. Therefore, this paper delves into the architecture and capabilities of the EN-HPCG grid using two different workload management systems: (i) Slurm (Open-Source) and (ii) PBS Pro (Licensed). Additionally, it assesses the grid's performance in specific high-throughput computing (HTC) applications using the NAS Grid parallel benchmark (NGB). The evaluation includes grid-level performance metrics such as throughput, and the number of tasks completed as a function of time. Also, the presented methodology aims to assist potential partners in their decision-making process to join the EN-HPCG grid, with a focus on the site speed-up metric. Our results showed that it is not advisable to integrate a cluster with high-speed hardware with a cluster possessing outdated hardware when using the Slurm scheduler. In contrast, the PBS Pro scheduler takes into account online decision-making in a dynamic environment using a unified grid.