the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
PSyclone 3: A source-to-source Fortran compiler for developing maintainable and performance-portable HPC applications
Abstract. PSyclone is a source-to-source Fortran compiler designed to programmatically optimize, parallelize, and instrument HPC applications via user-provided transformation scripts. These scripts allow for a clear separation of concerns between the scientific model, described in Fortran, and the optimization choices. This separation improves the maintainability of complex scientific applications by enabling independent exploration and development of each aspect. In addition, it provides a solution to achieve better performance portability for Fortran applications by encoding the transformations that are beneficial to each platform and compiler in different transformation scripts.
PSyclone supports two modes of operation. The first mode optimizes existing source code, including MPI-based applications, by making the necessary code transformations to effectively use the capabilities and programming models supported by each CPU and GPU vendor. This approach is demonstrated using a benchmark extracted from the NEMO ocean model. The second mode defines a kernel-based parallelism model with domain-specific Fortran-embedded metadata. This approach enables a stricter separation of concerns, thereby allowing PSyclone to take full control of the data dependencies and iteration spaces in order to improve its capabilities and generate distributed and shared-memory parallelism. This approach has been co-designed with the Met Office and is used in the Finite-Element based dynamical core of the LFRic atmospheric model.
Competing interests: David Ham (editor) was an investigator on the original Gung-Ho project that saw the inception of the PSyclone software described in this work.
Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this paper. While Copernicus Publications makes every effort to include appropriate place names, the final responsibility lies with the authors. Views expressed in the text are those of the authors and do not necessarily reflect the views of the publisher.- Preprint
(706 KB) - Metadata XML
- BibTeX
- EndNote
Status: final response (author comments only)
-
RC1: 'Comment on egusphere-2026-3844', Anonymous Referee #1, 16 Sep 2026
-
AC1: 'Reply on RC1', Sergi Siso, 12 Nov 2026
We thank the reviewer for taking the time to read and feedback on our paper. We respond inline to the various points below.
> One suggestion here, although I appreciate it would be more difficult to do, would be to compare and contrast performance generated by PSyclone compared to some of the other tools described in Section 2. The evaluation only compares PSyclone against itself, and this would strengthen things, for instance in Section 4.1 do a version of the NEMO Tracer-Advection Benchmark which manually has OpenMP target offload and/or OpenACC and compare the two (maybe performance and LoC between them). But I think this is fine for a follow on paper.
The aim of this paper is to position PSyclone as a tool for separating the domain science from optimisation logic. The absolute performance achieved will depend upon the contents of the transformation script. In this specific case (Section 4.1), PSyclone is programmatically replicating exactly what an HPC expert would do manually. We have therefore not compared to manual implementations since, if one of them performed better, the same code changes could reasonably be applied to the other one.
Similarly, for a given version (OpenMP or OpenACC), the LoCs produced by PSyclone would be the same as the manual version. However, one would then have a different version of the benchmark for each programming model/target as opposed to a different version of the PSyclone optimisation script. For this small benchmark, that is obviously not much of an advantage. The benefit of PSyclone's programmatic transformation approach is primarily seen in large code bases, where a manual implementation is costly to develop and maintain. In this context, the LoC difference is grounded in the size of the real software rather than an isolated benchmark. We intend to publish these results for the full NEMO model separately, as it requires significantly more detail for each NEMO component and includes authors who contributed to the NEMO PSyclone transformation effort but not specifically to PSyclone development.
We have extended the text to make these points clear.
> Mention of MLIR in section 2 is a bit naive, although that framework is written in C++ (like PSyclone is written in Python) it is compatible with Fortran, indeed Flang is built atop it.
We have clarified that what we meant was the difficulty for non-compiler experts to contribute to the MLIR dialects and we acknowledge the LLVM Fortran frontend and its applicability to Fortran codebases. We also extended the text to highlight the benefits of our approach.
> It would be good to present absolute numbers (e.g. runtime or GPt/s) in section 4.1, as it is currently quite difficult to interpret Figure 3.
We added absolute numbers in GPt/s as labels to the plot, while keeping the figure as a bar-plot of normalized bandwidth measurement to focus on portability across platforms.
> PSyclone generates OpenMP and OpenACC rather than CUDA or ROCm. It would be useful to add a sentence to explain why this is the case. It links back to my second point, as the reader I am left wondering if there is any performance left on the table for the GPU due to this decision.
We added a bullet point in the Code transformation limitations sections to expand on this comparison and provided a reference to a previous report that we published comparing psyclone inserted directives to CUDA (generated through DAWN).
Citation: https://doi.org/10.5194/egusphere-2026-3844-AC1
-
AC1: 'Reply on RC1', Sergi Siso, 12 Nov 2026
-
RC2: 'Comment on egusphere-2026-3844', Anonymous Referee #2, 07 Oct 2026
The paper describes the PSyclone compiler and workflow for optimizing atmospheric models written in Fortran. PSyclone is a source-to-source compiler that can accept general Fortran code and DSLs such as the LFRic DSL. The authors describe the Intermediate Representation and transformations from a bird’s eye view, with novel transformations that exchange communication for computation and introduce concurrency. When combined with a DSL, PSyclone demonstrates modest (2-8%) performance improvements and a 31% reduction in the number of communication calls on the LFRic dynamical core.
It is not an easy feat to write a compiler infrastructure that works at the scale of a dynamical core (compiling the entire codebase, including I/O using Codeblocks) and attain performance improvements over existing codes. It demonstrates that introducing redundant computations and having a halo-exchange-aware system is beneficial, which is a novel contribution over other prior works. However, the manuscript in its current state could be improved on two fronts: motivation and evaluation.
The first front is the reasoning for choosing the specific PSyclone approach and a description of its components that would motivate its use. The manuscript contains a brief high-level description of the IR, but it does not explain what the IR nodes are. Not necessarily as a table of operations (though that might be helpful), but in the sense of what computational patterns can or cannot be supported. The same applies for the PSyIR transformations. The descriptions of LFRic transformations in Section 5.2 are great and could be a basis for that. Particularly the Codeblock idea is commendable and novel, and has the power to potentially ingest any HPC Fortran application, but it is unclear what is in the analyzable regime. Moreover, for the “HaloExchange” node described in line 165, does “extensible” mean that it is currently implemented or subject to future work? A full read of the paper suggests that a version of it exists in the DSL level, but it is not adequately explained in the PsyIR part of the work.
The question of the extent of support extends to PSyKAl: is code outside the dynamical core sufficiently analyzed? Other physics modules are notoriously challenging to represent and optimize.
In terms of motivating the approach and compiler framework, I believe the work is well-positioned among the other competitors, but that should be explained better in the manuscript. Section 2 outlines the other approaches, but does not provide a comparative analysis. One crucial question that arises is what is the benefit of using a source-to-source Fortran compiler rather than writing MLIR passes? With the latest developments in flang, could PSyIR not be a part of HLFIR/FIR? How does the work relate to the MLIR stencil dialect? (Gysi et al., “Domain-specific multi-level IR rewriting for GPU: The Open Earth compiler for GPU-accelerated climate simulation”, ACM Transactions on Architecture and Code Optimization, Vol. 18, No. 4) For example, an equivalent or variant of “ArrayAssignment2LoopTrans” should already be performed during HLFIR-to-FIR lowering. While I agree with some of the software development claims in Section 2, they should be qualified with a specific example.
As for the other DSLs and Fortran parsers, what transformations or code-ingestion capabilities make PSyclone unique over those other approaches?
The second front in which the paper can improve is evaluation. The manuscript currently only makes self-comparisons with PSyclone. While difficult to compare with other DSLs such as GT4Py, any benchmark with both a DSL and Fortran source code will shed light onto the efficacy of the approach. If comparisons with other DSLs are impossible, what about the other Fortran compiler frameworks, such as Loki and the DACE Fortran parser? More importantly, Section 4.1 is missing comparisons with the reference NEMO codebase and OpenMP code (or GPU/OpenACC, if available).
As for portability evaluation, the central point that Section 5 makes is reducing performance portability fragility with DSLs. This merits some presentation of results over GPUs. However, only CPU results are presented for the GungHo dynamical core.
If these two aspects of the papers can be improved, it would make a much stronger argument as a whole.
Specific comments follow:
Section 4.1: You might be overcounting the memory bandwidth, as multiple contiguous accesses (as occur in stencils) may already be cached. Please use performance counters to measure the actual memory transactions. Also, would STREAM Copy rather than Triad give a more accurate bound on measured memory bandwidth? Figure 3 could also benefit from raw runtime numbers.
Section 4.1: Based on the comments in Section 4.2, there may be stark differences in performance based on the structure (e.g., loop schedule) of the code. Are the schedules different between the CPU and GPU versions, or is the source code exactly the same? Namely, is it an unmodified version of the original NEMO code?
Section 5.2: MatMulToCodeTrans- will you not gain more from vendor BLAS libraries such as MKL/OpenBLAS/cuBLAS/rocBLAS?
Section 6: “The application of PSyclone to transform existing models such as NEMO will be the subject of a future publication.” - Is this not the goal of Section 4?
Technical corrections:line 151: “given” -> “giving”
line 323: “discreet H100” -> “discrete H100”
line 324: Shouldn’t the H100 have a compute capability of 90 rather than 70?
Runtimes should be relatively stable but please include error bars.
Typographical: the paper combines both American and British English spelling.Citation: https://doi.org/10.5194/egusphere-2026-3844-RC2 -
AC2: 'Reply on RC2', Sergi Siso, 12 Nov 2026
We thank the reviewer for taking the time to read and feedback on our paper. We respond inline to the various points below.
> The paper describes the PSyclone compiler and workflow for optimizing atmospheric models written in Fortran. PSyclone is a source-to-source compiler that can accept general Fortran code and DSLs such as the LFRic DSL. The authors describe the Intermediate Representation and transformations from a bird’s eye view, with novel transformations that exchange communication for computation and introduce concurrency. When combined with a DSL, PSyclone demonstrates modest (2-8%) performance improvements and a 31% reduction in the number of communication calls on the LFRic dynamical core.
We thank the reviewer for this summary. For clarity, PSyclone implements the PSyKAl programming model and provides code generation and transformations for the LFRic DSL. LFRic developers write largely serial code, as kernels or algorithm components, and PSyclone generates the distributed-memory parallelization for those kernels.
The quoted performance improvements arise from applying selected user-script transformations to an implementation whose distributed-memory parallelization is already generated by PSyclone. So, in the LFRic case, we also demonstrate the ability of PSyclone to insert distributed-memory halo_exchange logic in addition to the ability of PSyclone to apply user-scripts to tune the performance for the given platform. We have updated the introductory text in the LFRic DSL section to clarify this.
> The manuscript contains a brief high-level description of the IR, but it does not explain what the IR nodes are. Not necessarily as a table of operations (though that might be helpful), but in the sense of what computational patterns can or cannot be supported. The same applies for the PSyIR transformations. The descriptions of LFRic transformations in Section 5.2 are great and could be a basis for that. Particularly the Codeblock idea is commendable and novel, and has the power to potentially ingest any HPC Fortran application, but it is unclear what is in the analyzable regime.
We agree that the description of the IR was somewhat terse. We have expanded the text to make clear what PSyIR can and cannot represent. At the time of writing, PSyIR has more than 100 nodes and 80 transformations implemented. We have summarised some of the key families in the text and added a reference to the User Guide where full documentation is available.
We have also added Codeblock coverage statistics for the full NEMO and LFRic Apps code repositories.
> Moreover, for the “HaloExchange” node described in line 165, does “extensible” mean that it is currently implemented or subject to future work? A full read of the paper suggests that a version of it exists in the DSL level, but it is not adequately explained in the PsyIR part of the work.
We have extended the description of the LFRic DSL with an example plus a brief description of each of the DSL node types. This makes it clear that we do have concrete implementations of them. By “extensible” we mean that different DSLs can extend the basic nodes as they need (while remaining compatible with the language nodes), with LFRic just being an example of it.
> The question of the extent of support extends to PSyKAl: is code outside the dynamical core sufficiently analyzed? Other physics modules are notoriously challenging to represent and optimize.
We have added a new section, “Adopting PSyclone with large code bases”. We describe the Met Office’s use of PSyclone outside the dynamical core and the way in which this is interfaced to the DSL.
> In terms of motivating the approach and compiler framework, I believe the work is well-positioned among the other competitors, but that should be explained better in the manuscript. Section 2 outlines the other approaches, but does not provide a comparative analysis. One crucial question that arises is what is the benefit of using a source-to-source Fortran compiler rather than writing MLIR passes? With the latest developments in flang, could PSyIR not be a part of HLFIR/FIR? How does the work relate to the MLIR stencil dialect? (Gysi et al., “Domain-specific multi-level IR rewriting for GPU: The Open Earth compiler for GPU-accelerated climate simulation”, ACM Transactions on Architecture and Code Optimization, Vol. 18, No. 4) For example, an equivalent or variant of “ArrayAssignment2LoopTrans” should already be performed during HLFIR-to-FIR lowering. While I agree with some of the software development claims in Section 2, they should be qualified with a specific example. As for the other DSLs and Fortran parsers, what transformations or code-ingestion capabilities make PSyclone unique over those other approaches?
We agree our comparison with MLIR developments was lacking and we have extended it. We also extended the last part of the section to provide a more comparative analysis of psyclone to the other solutions, including MLIR, DSLs and other source-to-source tools.
> The second front in which the paper can improve is evaluation. The manuscript currently only makes self-comparisons with PSyclone. While difficult to compare with other DSLs such as GT4Py, any benchmark with both a DSL and Fortran source code will shed light onto the efficacy of the approach. If comparisons with other DSLs are impossible, what about the other Fortran compiler frameworks, such as Loki and the DACE Fortran parser? More importantly, Section 4.1 is missing comparisons with the reference NEMO codebase and OpenMP code (or GPU/OpenACC, if available).
The focus of our evaluation is on the ability of PSyclone to apply multiple user-provided transformation to a single-source science specification, and therefore separating the domain logic from the optimisation logic. We do not claim that the transformation scripts are necessarily optimal. We updated the text in Section 4.1 to make this framing and limitations clearer.
In the past we published a report (ESIWACE2) using the same benchmark to compare the directives programmatically inserted by PSyclone with a CUDA implementation generated through DAWN. We agree this comparison is relevant and have cited the report and mentioned the performance comparison highlights.
> As for portability evaluation, the central point that Section 5 makes is reducing performance portability fragility with DSLs. This merits some presentation of results over GPUs. However, only CPU results are presented for the GungHo dynamical core.
We acknowledge that being able to demonstrate performance portability to GPUs with the LFRic DSL will be important. However, we feel that this is separate to the issue of fragility: for example, achieving a correct distributed memory implementation requires full coverage of the application. When working with a DSL, a developer needs to conform to the available constructs – anything else is rejected. Hence, if the DSL does not support a particular feature, this must be explicitly addressed – it cannot pass unnoticed.
The current production systems in the UK MetOffice are CPU-only, consequently most our efforts in LFRic have been on the CPU scripts. We added a new Section 6 where we mention that GPU-offloading scripts for LFRic are part of future work.
> Section 4.1: You might be overcounting the memory bandwidth, as multiple contiguous accesses (as occur in stencils) may already be cached. Please use performance counters to measure the actual memory transactions. Also, would STREAM Copy rather than Triad give a more accurate bound on measured memory bandwidth? Figure 3 could also benefit from raw runtime numbers.
Thanks for pointing out the overcounting issue. We updated Section 4.1 to acknowledge the overcounting and provided a performance counter measurement to validate that this effect is small. For the plot we kept the estimated bandwidth results because measured memory transactions could reflect implementation details rather than application performance. We also provide an explanation of why we believe STREAM Triad is a suitable characterisation of the benchmark memory pattern. As suggested, we also added labels with raw numbers (as GPt/s) to Figure 3.
> Section 4.1: Based on the comments in Section 4.2, there may be stark differences in performance based on the structure (e.g., loop schedule) of the code. Are the schedules different between the CPU and GPU versions, or is the source code exactly the same? Namely, is it an unmodified version of the original NEMO code?
The source code input to PSyclone is identical for all versions of the benchmark in 4.1, and the text has been slightly modified to clarify this.
> Section 5.2: MatMulToCodeTrans- will you not gain more from vendor BLAS libraries such as MKL/OpenBLAS/cuBLAS/rocBLAS?
This transformation is designed for very small matrix multiplications (the size is in the single digits). We found that some compilers manage to optimise across the intrinsic boundaries, but others treat them as a call and are not able to take advantage of loop optimisations with nearby loops, this limitation is gone applying the IntrinsicToCode transformation. In the past we explored specialised libraries like libxsmm, but the performance was not better than the inlined version. We have extended the text to indicate this.
> Section 6: “The application of PSyclone to transform existing models such as NEMO will be the subject of a future publication.” - Is this not the goal of Section 4?
Section 4 is only the tracer-advection component of NEMO – about 2% of the codebase. We used this part to showcase psyclone's code transformation approach with one of the main computationally expensive parts of NEMO. We intend to publish the full NEMO model work separately, as it requires significantly more detail for each different component (ocean,ice, biogeochemistry etc) of NEMO and includes authors who contributed to the NEMO PSyclone transformation effort but not specifically to PSyclone development.
> Technical corrections:
> line 151: “given” -> “giving”
> line 323: “discreet H100” -> “discrete H100”
> line 324: Shouldn’t the H100 have a compute capability of 90 rather than 70?
> Runtimes should be relatively stable but please include error bars.
> Typographical: the paper combines both American and British English spelling.
We fixed spelling and typographical errors. We left the PSyclone Transformation names with British spelling, as this is how they are named in PSyclone. The compute capability number was an error, we used `-gpu=ccnative` in all our run scripts, we updated the text accordingly. We tested adding error bars, but they were too small to be visible. We have updated the text to this effect.
Citation: https://doi.org/10.5194/egusphere-2026-3844-AC2
-
AC2: 'Reply on RC2', Sergi Siso, 12 Nov 2026
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 281 | 121 | 92 | 494 | 102 | 87 |
- HTML: 281
- PDF: 121
- XML: 92
- Total: 494
- BibTeX: 102
- EndNote: 87
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
This is a well written paper exploring a production tool in use across several codes. It presents an interesting approach to DSLs and code transformation, and flexible way of working across two different paradigms (direct transformation, and DSLs built around its PSyKAl abstraction). Whilst the paper is fairly heavy on the engineering side, this is worthwhile and a useful tool by the community that others will learn from and leverage.
One suggestion here, although I appreciate it would be more difficult to do, would be to compare and contrast performance generated by PSyclone compared to some of the other tools described in Section 2. The evaluation only compares PSyclone against itself, and this would strengthen things, for instance in Section 4.1 do a version of the NEMO Tracer-Advection Benchmark which manually has OpenMP target offload and/or OpenACC and compare the two (maybe performance and LoC between them). But I think this is fine for a follow on paper.
A few minor suggestions:
- Mention of MLIR in section 2 is a bit naive, although that framework is written in C++ (like PSyclone is written in Python) it is compatible with Fortran, indeed Flang is built atop it.
- It would be good to present absolute numbers (e.g. runtime or GPt/s) in section 4.1, as it is currently quite difficult to interpret Figure 3.
- PSyclone generates OpenMP and OpenACC rather than CUDA or ROCm. It would be useful to add a sentence to explain why this is the case. It links back to my second point, as the reader I am left wondering if there is any performance left on the table for the GPU due to this decision.