Configuring parallel use of custom ArcGIS toolboxes in a Linux high-performance computing environment

Abstract. High performance computing (HPC) clusters offer an abundance of processors, memory, and data storage that can help meet growing computational demands for large geospatial analyses. Esri’s proprietary ArcGIS toolboxes are a common way to develop, document, and share geospatial code; however, these toolboxes are typically developed for use in Esri’s ArcGIS Pro for Windows which limits their use in Linux, the dominant HPC operating system. Esri’s support of Linux for their server software opens the possibility of using ArcGIS toolboxes in an HPC environment, but use of this software proved problematic for concurrent, parallel processing offering little benefit over a single workstation. We developed a solution using container technology allowing us to run hundreds of concurrent tasks that use ArcGIS toolboxes. This approach is designed to take advantage of the many existing ArcGIS toolboxes and use them in parallel for workflows that can be reasonably divided into independent sub-tasks. With careful setup and use of an HPC job scheduler, we reduced total processing time for one analysis 95.6 % by dividing the analysis into 202 sub-tasks. This solution allows other HPC users to take advantage of parallel use of ArcGIS Toolboxes, avoid the effort of translating existing workflows to Linux native software, and establish a bridge between desktop GIS and HPC systems.