the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 16 Feb 2024
Physics-motivated Cell-octree Adaptive Mesh Refinement in the Vlasiator 5.3 Global Hybrid-Vlasov Code
Abstract. Automatically adaptive grid resolution is a common way of improving simulation accuracy while keeping the computational efficiency at a manageable level. In space physics adaptive grid strategies are especially useful as simulation volumes are extreme, while the most accurate physical description is based on electron dynamics and hence requires very small grid cells and time steps. Therefore, many past global simulations encompassing e.g. the near-Earth space have made tradeoffs in terms of the physical description and used laws of magnetohydrodynamics (MHD) that require less accurate grid resolutions. Recently, using supercomputers, it has become possible to model the near-Earth space domain with an ion-hybrid scheme going beyond the MHD-based fluid dynamics. These simulations, however, must develop a new adaptive mesh strategy beyond what is used in MHD simulations.
We developed an automatically adaptive grid refinement strategy for ion-hybrid Vlasov schemes, and implemented it within the Vlasiator global solar wind – magnetosphere – ionosphere simulation Vlasiator. This method automatically adapts the resolution of the Vlasiator grid using two indices: one formed as a maximum of dimensionless gradients measuring the rate of spatial change in selected variables, and the other derived from the ratio of the current density to the magnetic field density perpendicular to the current. Both these indices can be tuned independently to reach a desired level of refinement and computational load. We test the indices independently and compare the results to a control run using static refinement.
The results show that adaptive refinement highlights relevant regions of the simulation domain and keeps the computational effort at a manageable level. We find that the refinement shows some overhead in rate of cells solved per second. This overhead can be large compared to the control run without adaptive refinement, possibly due to resource utilisation, grid complexity and issues in load balancing. These issues lay a development roadmap for future optimisations.
-
Notice on discussion status
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
-
Preprint
(1816 KB)
-
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(1816 KB) - Metadata XML
- BibTeX
- EndNote
- Final revised paper
Journal article(s) based on this preprint
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2024-301', Anonymous Referee #1, 16 Apr 2024
Line 64: Does Vlasiator use the real mass ratio?
Line 70: Please elaborate more about what spatial optimizations are needed.
Line 84: AMR is not used in MHD-AEPIC. MHD-AEPIC is dynamically apply kinetic physics in regions that are needed. The grid resolution is not varying in space. BATS-R-US indeed uses AMR in the MHD grid. I suggest removing the MHD-AEPIC citation here.
Figure 5(a): Why the cell number increases for the unrefined run?
Table 1: Please list the total runtime of each run in the table, that would justify using AMR in the future production runs, even the scaling seems to be bad.
Citation: https://doi.org/10.5194/egusphere-2024-301-RC1 -
AC1: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
We thank the referees for their comments and feedback, addressed below.
RC1:
> Line 64: Does Vlasiator use the real mass ratio?
Vlasiator uses the real mass ratio for protons and electrons since in the hybrid model electrons aren’t explicitly modelled, obviating the need for a non-physical mass ratio. Real electron masses are used in the electron pressure gradient term of Ohm’s law.
> Line 70: Please elaborate more about what spatial optimizations are needed.
The further spatial optimizations on line 70 are in reference to mesh refinement introduced in the following sub-section. We will clarify this in the revision.
> Line 84: AMR is not used in MHD-AEPIC. MHD-AEPIC is dynamically apply kinetic physics in regions that are needed. The grid resolution is not varying in space. BATS-R-US indeed uses AMR in the MHD grid. I suggest removing the MHD-AEPIC citation here.
MHD-AEPIC citation will be removed from line 84, thank you for the correction
> Figure 5(a): Why the cell number increases for the unrefined run?
In figures 5a and 5b the plotted cell count is the entire phase-space i.e. including velocity space which fills out as the state evolves. It is a measure of total computational/memory load. We will amend the figure to clarify this.
> Table 1: Please list the total runtime of each run in the table, that would justify using AMR in the future production runs, even the scaling seems to be bad.
We will provide total runtime in Table 3.
Citation: https://doi.org/10.5194/egusphere-2024-301-AC1 -
AC4: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
References:
Dubart, M., Ganse, U., Osmane, A., Johlander, A., Battarbee, M., Grandin, M., Pfau-Kempf, Y., Turc, L., & Palmroth, M. (2020). Resolution dependence of magnetosheath waves in global hybrid-Vlasov simulations. Annales Geophysicae, 38(6), 1283–1298. https://doi.org/10.5194/angeo-38-1283-2020
Ganse, U., Koskela, T., Battarbee, M., Pfau-Kempf, Y., Papadakis, K., Alho, M., Bussov, M., Cozzani, G., Dubart, M., George, H., Gordeev, E., Grandin, M., Horaites, K., Suni, J., Tarvus, V., Kebede, F. T., Turc, L., Zhou, H., & Palmroth, M. (2023). Enabling technology for global 3D + 3V hybrid-Vlasov simulations of near-Earth space. Physics of Plasmas, 30(4), 042902. https://doi.org/10.1063/5.0134387
Rembiasz, T., Obergaulinger, M., Cerdá-Durán, P., Aloy, M.-Á., & Müller, E. (2017). On the Measurements of Numerical Viscosity and Resistivity in Eulerian MHD Codes. The Astrophysical Journal Supplement Series, 230(2), 18. https://doi.org/10.3847/1538-4365/aa6254
Citation: https://doi.org/10.5194/egusphere-2024-301-AC4
-
AC1: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
-
RC2: 'Comment on egusphere-2024-301', Anonymous Referee #2, 16 Apr 2024
I have read the manuscript "Physics-motivated Cell-octree Adaptive Mesh
Refinement in the Vlasiator 5.3 Global Hybrid-Vlasov Code" authored by
Kotipalo et al. The manuscript introduces a new adaptive mesh refinement
strategy in the kinetic hybrid-Vlasov code, Vlasiator. A series of test runs
were carried to compare the new method with the existing static mesh
refinement approach. Although, the performance of the new method is not
perfect, the overall results are interesting, and the new strategy warrants
further investigation. Beyond some minor comments, I would recommend that
this paper be accepted for publication in GMD.Was OpenMP used in the test runs? Regardless of the answer, could the
authors provide additional discussion on its impact on performance,
particularly in terms of load balancing?It is speculated that the load balance was non-ideal because the weights
used in the load balancing algorithm did not account for changes in the
mesh. Would more frequent load balancing help mitigate this issue?I noticed in the Acknowledgements that the simulations were performed on CSC
Mahti supercomputer. However, while reading the manuscript, I had questions
about this. It would help the reader if a brief description of the computer
were included alongside the presentation of the test results.Line 138, starting with 'These gradients are': Technically speaking, an
expression like (delta B)^2 / (2 mu_0) is not the gradient of the magnetic
field energy, which would be B delta B / mu_0. To avoid confusion, could the
authors revise this sentence?There is a typo in Line 173. `his` -> `this`?
Citation: https://doi.org/10.5194/egusphere-2024-301-RC2 -
AC2: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
RC2:
> Was OpenMP used in the test runs? Regardless of the answer, could the authors provide additional discussion on its impact on performance, particularly in terms of load balancing?
OpenMP was used in all tests. Each process ran on two physical cores with hyperthreading, for a total of four OpenMP threads per process. Task balance was not the principal factor investigated, so every run used the same amount of cores per task. We will add the used threading settings in the revision.
> It is speculated that the load balance was non-ideal because the weights used in the load balancing algorithm did not account for changes in the mesh. Would more frequent load balancing help mitigate this issue?
More frequent load balancing is expected to help with AMR performance, or rather, not refining before every load balance. The intention was to “stress-test” AMR to see the overhead of refinement itself on maximal cadence in comparison to load balance. Load balancing itself also incurs a cost, so too frequent load balances will also be inefficient.
> I noticed in the Acknowledgements that the simulations were performed on CSC Mahti supercomputer. However, while reading the manuscript, I had questions about this. It would help the reader if a brief description of the computer were included alongside the presentation of the test results
Detailed information of the Mahti supercomputer is available in the Mahti documentation (https://docs.csc.fi/computing/systems-mahti/) – in these tests the CPU nodes were used. We will include a description in the revision.
> Line 138, starting with 'These gradients are': Technically speaking, an expression like (delta B)^2 / (2 mu_0) is not the gradient of the magnetic field energy, which would be B delta B / mu_0. To avoid confusion, could the authors revise this sentence?
> There is a typo in Line 173. `his` -> `this`?Thank you for the observant correction. The mention of gradients and the typo will be corrected in the revision.
Citation: https://doi.org/10.5194/egusphere-2024-301-AC2 -
AC4: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
References:
Dubart, M., Ganse, U., Osmane, A., Johlander, A., Battarbee, M., Grandin, M., Pfau-Kempf, Y., Turc, L., & Palmroth, M. (2020). Resolution dependence of magnetosheath waves in global hybrid-Vlasov simulations. Annales Geophysicae, 38(6), 1283–1298. https://doi.org/10.5194/angeo-38-1283-2020
Ganse, U., Koskela, T., Battarbee, M., Pfau-Kempf, Y., Papadakis, K., Alho, M., Bussov, M., Cozzani, G., Dubart, M., George, H., Gordeev, E., Grandin, M., Horaites, K., Suni, J., Tarvus, V., Kebede, F. T., Turc, L., Zhou, H., & Palmroth, M. (2023). Enabling technology for global 3D + 3V hybrid-Vlasov simulations of near-Earth space. Physics of Plasmas, 30(4), 042902. https://doi.org/10.1063/5.0134387
Rembiasz, T., Obergaulinger, M., Cerdá-Durán, P., Aloy, M.-Á., & Müller, E. (2017). On the Measurements of Numerical Viscosity and Resistivity in Eulerian MHD Codes. The Astrophysical Journal Supplement Series, 230(2), 18. https://doi.org/10.3847/1538-4365/aa6254
Citation: https://doi.org/10.5194/egusphere-2024-301-AC4
-
AC2: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
-
RC3: 'Comment on egusphere-2024-301', Anonymous Referee #3, 17 Apr 2024
General comments
This paper describes the implementation of adaptive mesh refinement in the Vlasiator hybrid code. The authors use a series of metrics previously used in MHD codes to prescribe where refinement takes place, and perform a series of simulations, providing statistics on the computational efficiency. While the implementation is a useful contribution, and the computational sections are well-written, the paper requires revision before it can be published due to missing physics information.
Specific comments
1) Although this is primarily a numerical paper, there is a lack of physics information on the system being studied. This inculdes
- Line 52 - what is the ion kinetic scale compared to the resolution?
- What are the physical parameters of the simulation being used for the study (eg plasma and field parameters)2) Figure 1 - What are the units in the colour plot? Also, the physics appears different in the two runs (the most obvious difference is the flux rope in the tail)?
3) Line 239 - Should this be XY and XZ planes?
4) Line 245 - What do the authors mean when referring to "foreshocks"? These do not appear to be the perturbations upstream of the quasi-parallel shock, which is the usual definition.
5) Figure 5 - The bar charts in (c)-(j) have some quantities obscured due to the overlapping. I would suggest modifying the opacity or a line plot.
6) Figure 6 - It would be helpful to provide a plot of the control in addition to the existing plots.
7) In general, the authors do not show that their simulations converge. This is alluded to in point 2), where the behaviour of the magnetotail looks different in the two runs. Without this information, whether the method works cannot be judged.
8) Table 3 - I suggest providing the total run time in the table as well, since that is of practical use.
Suggestions
1) Line 179 - Please define induced refinement.
2) I would suggest using a different font/italics for dccrg
Technical corrections1) Line 138 - move (a-e) to before the quantities. E.g (a) particle density ....
2) Line 173 - his -> this
3) There are certain parts of the text where contractions are used (e.g don't). These do not fit stylistically with the rest of the paper.Citation: https://doi.org/10.5194/egusphere-2024-301-RC3 -
AC3: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
RC3:
> Line 52 - what is the ion kinetic scale compared to the resolution?
The thermal Larmor radius (~130 km) or ion inertial length (~230 km) are not resolved in these runs, where the resolution is 2000 km on the finest level. As also discussed by Ganse et al. (2023), even 1000 km currently achieved at production scale is a compromise. Performing several case studies at production run resolution would require significant computational resources through one or more dedicated tier-0 resource applications. We will include this discussion point in the paper.
> What are the physical parameters of the simulation being used for the study (eg plasma and field parameters)
Thank you for pointing out this omission. The simulation was initialised with a solar wind proton temperature of 5 * 105 K, proton number density of 1/cm^3, and a solar wind velocity of -750 km/s in the x-direction. Within 30 * 103 km from the origin, the plasma is at rest, with velocity tapering sinusoidally from a distance of 30 * 103 to a distance of 100 * 103 km, from where the full solar wind velocity is used. The background field consists of -5 nT IMF flux in the z-direction and a vector dipole field equivalent to Earth’s. We will provide the physical setup in the revision.
> Figure 1 - What are the units in the colour plot? Also, the physics appears different in the two runs (the most obvious difference is the flux rope in the tail)?
Units in Figure 1 are 1/m^3, we will add this to the revision. Physical differences stem from the lower run only having run for 50s with AMR, with kinetic effects remaining unresolved during the 500s initialization on the lower resolution. Thus, the unresolved 500 second initialization period allows global but not kinetic features to form, and the full resolution should be allowed to impact the results for several ion gyroperiods before using the data for scientific analysis.
> Line 239 - Should this be XY and XZ planes?
Yes, we will correct this in the revision.
> Line 245 - What do the authors mean when referring to "foreshocks"? These do not appear to be the perturbations upstream of the quasi-parallel shock, which is the usual definition.
We acknowledge that it is unusual to refer to the structures in Figure 6 as foreshocks. However, with the Bz IMF, the flanks do in fact build up to a quasi-parallel shock, even though the shock-normal plasma flow is miniscule. From a point of view of particle shock physics, we suggest that this region is still best described as a foreshock, even if the upstream plasma flows more in parallel with the shock front than directed towards it.
> Figure 5 - The bar charts in (c)-(j) have some quantities obscured due to the overlapping. I would suggest modifying the opacity or a line plot.
The histograms in Figure 5 (c)-(j) are stacked, i.e. the total height of the bars show the portion of spatial volume with some value of alpha, with the heights of the coloured sections of the bars showing the portion of cells on each refinement level. We apologise for the confusion and will clarify this in the revision.
> Figure 6 - It would be helpful to provide a plot of the control in addition to the existing plots
Thank you for the suggestion. Plots of the control run are given in Figure 2, and we will replicate them in Figure 6 for easier comparison.
> In general, the authors do not show that their simulations converge. This is alluded to in point 2), where the behaviour of the magnetotail looks different in the two runs. Without this information, whether the method works cannot be judged.
Thank you for raising an interesting point. The physics observed in the control and AMR runs are different, as for example the lower run in Figure 1 has only been refined in the tail for 50 seconds. Indeed the simulations haven’t converged and in an absolute sense cannot converge – different local resolutions affect physical phenomena such as reconnection rate through numerical resistivity (Rembiasz et al., 2017). It is also known that some kinetic phenomena cannot be described unless sufficient resolution is achieved (Dubart et al, 2020). Running longer with AMR enabled is expected to result in more qualitatively similar structures; the short 50 s period of refinement was meant to determine the shape of refined regions, thus ascertaining the relevance of the chosen refinement parameters.
> Table 3 - I suggest providing the total run time in the table as well, since that is of practical use.
We will provide this in Table 3.
> Line 179 - Please define induced refinement.
Thank you for the clarification request. Induced refinement is dccrg behaviour for enforcing maximum refinement differences between neighbours. Essentially, refining a cell whose neighbours are too coarse refines those neighbours. We will include this additional information in the revision.
> I would suggest using a different font/italics for dccrg
The authors acknowledge that the lowercase initialism dccrg is less than ideal for readability. We will capitalise it in the revision.
> Line 138 - move (a-e) to before the quantities. E.g (a) particle density ....
> Line 173 - his -> this
> There are certain parts of the text where contractions are used (e.g don't). These do not fit stylistically with the rest of the paper.We will make these technical corrections in the revision.
Citation: https://doi.org/10.5194/egusphere-2024-301-AC3 -
AC4: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
References:
Dubart, M., Ganse, U., Osmane, A., Johlander, A., Battarbee, M., Grandin, M., Pfau-Kempf, Y., Turc, L., & Palmroth, M. (2020). Resolution dependence of magnetosheath waves in global hybrid-Vlasov simulations. Annales Geophysicae, 38(6), 1283–1298. https://doi.org/10.5194/angeo-38-1283-2020
Ganse, U., Koskela, T., Battarbee, M., Pfau-Kempf, Y., Papadakis, K., Alho, M., Bussov, M., Cozzani, G., Dubart, M., George, H., Gordeev, E., Grandin, M., Horaites, K., Suni, J., Tarvus, V., Kebede, F. T., Turc, L., Zhou, H., & Palmroth, M. (2023). Enabling technology for global 3D + 3V hybrid-Vlasov simulations of near-Earth space. Physics of Plasmas, 30(4), 042902. https://doi.org/10.1063/5.0134387
Rembiasz, T., Obergaulinger, M., Cerdá-Durán, P., Aloy, M.-Á., & Müller, E. (2017). On the Measurements of Numerical Viscosity and Resistivity in Eulerian MHD Codes. The Astrophysical Journal Supplement Series, 230(2), 18. https://doi.org/10.3847/1538-4365/aa6254
Citation: https://doi.org/10.5194/egusphere-2024-301-AC4
-
AC3: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
-
AC1: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
We thank the referees for their comments and feedback, addressed below.
RC1:
> Line 64: Does Vlasiator use the real mass ratio?
Vlasiator uses the real mass ratio for protons and electrons since in the hybrid model electrons aren’t explicitly modelled, obviating the need for a non-physical mass ratio. Real electron masses are used in the electron pressure gradient term of Ohm’s law.
> Line 70: Please elaborate more about what spatial optimizations are needed.
The further spatial optimizations on line 70 are in reference to mesh refinement introduced in the following sub-section. We will clarify this in the revision.
> Line 84: AMR is not used in MHD-AEPIC. MHD-AEPIC is dynamically apply kinetic physics in regions that are needed. The grid resolution is not varying in space. BATS-R-US indeed uses AMR in the MHD grid. I suggest removing the MHD-AEPIC citation here.
MHD-AEPIC citation will be removed from line 84, thank you for the correction
> Figure 5(a): Why the cell number increases for the unrefined run?
In figures 5a and 5b the plotted cell count is the entire phase-space i.e. including velocity space which fills out as the state evolves. It is a measure of total computational/memory load. We will amend the figure to clarify this.
> Table 1: Please list the total runtime of each run in the table, that would justify using AMR in the future production runs, even the scaling seems to be bad.
We will provide total runtime in Table 3.
Citation: https://doi.org/10.5194/egusphere-2024-301-AC1 -
AC2: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
RC2:
> Was OpenMP used in the test runs? Regardless of the answer, could the authors provide additional discussion on its impact on performance, particularly in terms of load balancing?
OpenMP was used in all tests. Each process ran on two physical cores with hyperthreading, for a total of four OpenMP threads per process. Task balance was not the principal factor investigated, so every run used the same amount of cores per task. We will add the used threading settings in the revision.
> It is speculated that the load balance was non-ideal because the weights used in the load balancing algorithm did not account for changes in the mesh. Would more frequent load balancing help mitigate this issue?
More frequent load balancing is expected to help with AMR performance, or rather, not refining before every load balance. The intention was to “stress-test” AMR to see the overhead of refinement itself on maximal cadence in comparison to load balance. Load balancing itself also incurs a cost, so too frequent load balances will also be inefficient.
> I noticed in the Acknowledgements that the simulations were performed on CSC Mahti supercomputer. However, while reading the manuscript, I had questions about this. It would help the reader if a brief description of the computer were included alongside the presentation of the test results
Detailed information of the Mahti supercomputer is available in the Mahti documentation (https://docs.csc.fi/computing/systems-mahti/) – in these tests the CPU nodes were used. We will include a description in the revision.
> Line 138, starting with 'These gradients are': Technically speaking, an expression like (delta B)^2 / (2 mu_0) is not the gradient of the magnetic field energy, which would be B delta B / mu_0. To avoid confusion, could the authors revise this sentence?
> There is a typo in Line 173. `his` -> `this`?Thank you for the observant correction. The mention of gradients and the typo will be corrected in the revision.
Citation: https://doi.org/10.5194/egusphere-2024-301-AC2 -
AC3: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
RC3:
> Line 52 - what is the ion kinetic scale compared to the resolution?
The thermal Larmor radius (~130 km) or ion inertial length (~230 km) are not resolved in these runs, where the resolution is 2000 km on the finest level. As also discussed by Ganse et al. (2023), even 1000 km currently achieved at production scale is a compromise. Performing several case studies at production run resolution would require significant computational resources through one or more dedicated tier-0 resource applications. We will include this discussion point in the paper.
> What are the physical parameters of the simulation being used for the study (eg plasma and field parameters)
Thank you for pointing out this omission. The simulation was initialised with a solar wind proton temperature of 5 * 105 K, proton number density of 1/cm^3, and a solar wind velocity of -750 km/s in the x-direction. Within 30 * 103 km from the origin, the plasma is at rest, with velocity tapering sinusoidally from a distance of 30 * 103 to a distance of 100 * 103 km, from where the full solar wind velocity is used. The background field consists of -5 nT IMF flux in the z-direction and a vector dipole field equivalent to Earth’s. We will provide the physical setup in the revision.
> Figure 1 - What are the units in the colour plot? Also, the physics appears different in the two runs (the most obvious difference is the flux rope in the tail)?
Units in Figure 1 are 1/m^3, we will add this to the revision. Physical differences stem from the lower run only having run for 50s with AMR, with kinetic effects remaining unresolved during the 500s initialization on the lower resolution. Thus, the unresolved 500 second initialization period allows global but not kinetic features to form, and the full resolution should be allowed to impact the results for several ion gyroperiods before using the data for scientific analysis.
> Line 239 - Should this be XY and XZ planes?
Yes, we will correct this in the revision.
> Line 245 - What do the authors mean when referring to "foreshocks"? These do not appear to be the perturbations upstream of the quasi-parallel shock, which is the usual definition.
We acknowledge that it is unusual to refer to the structures in Figure 6 as foreshocks. However, with the Bz IMF, the flanks do in fact build up to a quasi-parallel shock, even though the shock-normal plasma flow is miniscule. From a point of view of particle shock physics, we suggest that this region is still best described as a foreshock, even if the upstream plasma flows more in parallel with the shock front than directed towards it.
> Figure 5 - The bar charts in (c)-(j) have some quantities obscured due to the overlapping. I would suggest modifying the opacity or a line plot.
The histograms in Figure 5 (c)-(j) are stacked, i.e. the total height of the bars show the portion of spatial volume with some value of alpha, with the heights of the coloured sections of the bars showing the portion of cells on each refinement level. We apologise for the confusion and will clarify this in the revision.
> Figure 6 - It would be helpful to provide a plot of the control in addition to the existing plots
Thank you for the suggestion. Plots of the control run are given in Figure 2, and we will replicate them in Figure 6 for easier comparison.
> In general, the authors do not show that their simulations converge. This is alluded to in point 2), where the behaviour of the magnetotail looks different in the two runs. Without this information, whether the method works cannot be judged.
Thank you for raising an interesting point. The physics observed in the control and AMR runs are different, as for example the lower run in Figure 1 has only been refined in the tail for 50 seconds. Indeed the simulations haven’t converged and in an absolute sense cannot converge – different local resolutions affect physical phenomena such as reconnection rate through numerical resistivity (Rembiasz et al., 2017). It is also known that some kinetic phenomena cannot be described unless sufficient resolution is achieved (Dubart et al, 2020). Running longer with AMR enabled is expected to result in more qualitatively similar structures; the short 50 s period of refinement was meant to determine the shape of refined regions, thus ascertaining the relevance of the chosen refinement parameters.
> Table 3 - I suggest providing the total run time in the table as well, since that is of practical use.
We will provide this in Table 3.
> Line 179 - Please define induced refinement.
Thank you for the clarification request. Induced refinement is dccrg behaviour for enforcing maximum refinement differences between neighbours. Essentially, refining a cell whose neighbours are too coarse refines those neighbours. We will include this additional information in the revision.
> I would suggest using a different font/italics for dccrg
The authors acknowledge that the lowercase initialism dccrg is less than ideal for readability. We will capitalise it in the revision.
> Line 138 - move (a-e) to before the quantities. E.g (a) particle density ....
> Line 173 - his -> this
> There are certain parts of the text where contractions are used (e.g don't). These do not fit stylistically with the rest of the paper.We will make these technical corrections in the revision.
Citation: https://doi.org/10.5194/egusphere-2024-301-AC3 -
AC4: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
References:
Dubart, M., Ganse, U., Osmane, A., Johlander, A., Battarbee, M., Grandin, M., Pfau-Kempf, Y., Turc, L., & Palmroth, M. (2020). Resolution dependence of magnetosheath waves in global hybrid-Vlasov simulations. Annales Geophysicae, 38(6), 1283–1298. https://doi.org/10.5194/angeo-38-1283-2020
Ganse, U., Koskela, T., Battarbee, M., Pfau-Kempf, Y., Papadakis, K., Alho, M., Bussov, M., Cozzani, G., Dubart, M., George, H., Gordeev, E., Grandin, M., Horaites, K., Suni, J., Tarvus, V., Kebede, F. T., Turc, L., Zhou, H., & Palmroth, M. (2023). Enabling technology for global 3D + 3V hybrid-Vlasov simulations of near-Earth space. Physics of Plasmas, 30(4), 042902. https://doi.org/10.1063/5.0134387
Rembiasz, T., Obergaulinger, M., Cerdá-Durán, P., Aloy, M.-Á., & Müller, E. (2017). On the Measurements of Numerical Viscosity and Resistivity in Eulerian MHD Codes. The Astrophysical Journal Supplement Series, 230(2), 18. https://doi.org/10.3847/1538-4365/aa6254
Citation: https://doi.org/10.5194/egusphere-2024-301-AC4
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2024-301', Anonymous Referee #1, 16 Apr 2024
Line 64: Does Vlasiator use the real mass ratio?
Line 70: Please elaborate more about what spatial optimizations are needed.
Line 84: AMR is not used in MHD-AEPIC. MHD-AEPIC is dynamically apply kinetic physics in regions that are needed. The grid resolution is not varying in space. BATS-R-US indeed uses AMR in the MHD grid. I suggest removing the MHD-AEPIC citation here.
Figure 5(a): Why the cell number increases for the unrefined run?
Table 1: Please list the total runtime of each run in the table, that would justify using AMR in the future production runs, even the scaling seems to be bad.
Citation: https://doi.org/10.5194/egusphere-2024-301-RC1 -
AC1: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
We thank the referees for their comments and feedback, addressed below.
RC1:
> Line 64: Does Vlasiator use the real mass ratio?
Vlasiator uses the real mass ratio for protons and electrons since in the hybrid model electrons aren’t explicitly modelled, obviating the need for a non-physical mass ratio. Real electron masses are used in the electron pressure gradient term of Ohm’s law.
> Line 70: Please elaborate more about what spatial optimizations are needed.
The further spatial optimizations on line 70 are in reference to mesh refinement introduced in the following sub-section. We will clarify this in the revision.
> Line 84: AMR is not used in MHD-AEPIC. MHD-AEPIC is dynamically apply kinetic physics in regions that are needed. The grid resolution is not varying in space. BATS-R-US indeed uses AMR in the MHD grid. I suggest removing the MHD-AEPIC citation here.
MHD-AEPIC citation will be removed from line 84, thank you for the correction
> Figure 5(a): Why the cell number increases for the unrefined run?
In figures 5a and 5b the plotted cell count is the entire phase-space i.e. including velocity space which fills out as the state evolves. It is a measure of total computational/memory load. We will amend the figure to clarify this.
> Table 1: Please list the total runtime of each run in the table, that would justify using AMR in the future production runs, even the scaling seems to be bad.
We will provide total runtime in Table 3.
Citation: https://doi.org/10.5194/egusphere-2024-301-AC1 -
AC4: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
References:
Dubart, M., Ganse, U., Osmane, A., Johlander, A., Battarbee, M., Grandin, M., Pfau-Kempf, Y., Turc, L., & Palmroth, M. (2020). Resolution dependence of magnetosheath waves in global hybrid-Vlasov simulations. Annales Geophysicae, 38(6), 1283–1298. https://doi.org/10.5194/angeo-38-1283-2020
Ganse, U., Koskela, T., Battarbee, M., Pfau-Kempf, Y., Papadakis, K., Alho, M., Bussov, M., Cozzani, G., Dubart, M., George, H., Gordeev, E., Grandin, M., Horaites, K., Suni, J., Tarvus, V., Kebede, F. T., Turc, L., Zhou, H., & Palmroth, M. (2023). Enabling technology for global 3D + 3V hybrid-Vlasov simulations of near-Earth space. Physics of Plasmas, 30(4), 042902. https://doi.org/10.1063/5.0134387
Rembiasz, T., Obergaulinger, M., Cerdá-Durán, P., Aloy, M.-Á., & Müller, E. (2017). On the Measurements of Numerical Viscosity and Resistivity in Eulerian MHD Codes. The Astrophysical Journal Supplement Series, 230(2), 18. https://doi.org/10.3847/1538-4365/aa6254
Citation: https://doi.org/10.5194/egusphere-2024-301-AC4
-
AC1: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
-
RC2: 'Comment on egusphere-2024-301', Anonymous Referee #2, 16 Apr 2024
I have read the manuscript "Physics-motivated Cell-octree Adaptive Mesh
Refinement in the Vlasiator 5.3 Global Hybrid-Vlasov Code" authored by
Kotipalo et al. The manuscript introduces a new adaptive mesh refinement
strategy in the kinetic hybrid-Vlasov code, Vlasiator. A series of test runs
were carried to compare the new method with the existing static mesh
refinement approach. Although, the performance of the new method is not
perfect, the overall results are interesting, and the new strategy warrants
further investigation. Beyond some minor comments, I would recommend that
this paper be accepted for publication in GMD.Was OpenMP used in the test runs? Regardless of the answer, could the
authors provide additional discussion on its impact on performance,
particularly in terms of load balancing?It is speculated that the load balance was non-ideal because the weights
used in the load balancing algorithm did not account for changes in the
mesh. Would more frequent load balancing help mitigate this issue?I noticed in the Acknowledgements that the simulations were performed on CSC
Mahti supercomputer. However, while reading the manuscript, I had questions
about this. It would help the reader if a brief description of the computer
were included alongside the presentation of the test results.Line 138, starting with 'These gradients are': Technically speaking, an
expression like (delta B)^2 / (2 mu_0) is not the gradient of the magnetic
field energy, which would be B delta B / mu_0. To avoid confusion, could the
authors revise this sentence?There is a typo in Line 173. `his` -> `this`?
Citation: https://doi.org/10.5194/egusphere-2024-301-RC2 -
AC2: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
RC2:
> Was OpenMP used in the test runs? Regardless of the answer, could the authors provide additional discussion on its impact on performance, particularly in terms of load balancing?
OpenMP was used in all tests. Each process ran on two physical cores with hyperthreading, for a total of four OpenMP threads per process. Task balance was not the principal factor investigated, so every run used the same amount of cores per task. We will add the used threading settings in the revision.
> It is speculated that the load balance was non-ideal because the weights used in the load balancing algorithm did not account for changes in the mesh. Would more frequent load balancing help mitigate this issue?
More frequent load balancing is expected to help with AMR performance, or rather, not refining before every load balance. The intention was to “stress-test” AMR to see the overhead of refinement itself on maximal cadence in comparison to load balance. Load balancing itself also incurs a cost, so too frequent load balances will also be inefficient.
> I noticed in the Acknowledgements that the simulations were performed on CSC Mahti supercomputer. However, while reading the manuscript, I had questions about this. It would help the reader if a brief description of the computer were included alongside the presentation of the test results
Detailed information of the Mahti supercomputer is available in the Mahti documentation (https://docs.csc.fi/computing/systems-mahti/) – in these tests the CPU nodes were used. We will include a description in the revision.
> Line 138, starting with 'These gradients are': Technically speaking, an expression like (delta B)^2 / (2 mu_0) is not the gradient of the magnetic field energy, which would be B delta B / mu_0. To avoid confusion, could the authors revise this sentence?
> There is a typo in Line 173. `his` -> `this`?Thank you for the observant correction. The mention of gradients and the typo will be corrected in the revision.
Citation: https://doi.org/10.5194/egusphere-2024-301-AC2 -
AC4: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
References:
Dubart, M., Ganse, U., Osmane, A., Johlander, A., Battarbee, M., Grandin, M., Pfau-Kempf, Y., Turc, L., & Palmroth, M. (2020). Resolution dependence of magnetosheath waves in global hybrid-Vlasov simulations. Annales Geophysicae, 38(6), 1283–1298. https://doi.org/10.5194/angeo-38-1283-2020
Ganse, U., Koskela, T., Battarbee, M., Pfau-Kempf, Y., Papadakis, K., Alho, M., Bussov, M., Cozzani, G., Dubart, M., George, H., Gordeev, E., Grandin, M., Horaites, K., Suni, J., Tarvus, V., Kebede, F. T., Turc, L., Zhou, H., & Palmroth, M. (2023). Enabling technology for global 3D + 3V hybrid-Vlasov simulations of near-Earth space. Physics of Plasmas, 30(4), 042902. https://doi.org/10.1063/5.0134387
Rembiasz, T., Obergaulinger, M., Cerdá-Durán, P., Aloy, M.-Á., & Müller, E. (2017). On the Measurements of Numerical Viscosity and Resistivity in Eulerian MHD Codes. The Astrophysical Journal Supplement Series, 230(2), 18. https://doi.org/10.3847/1538-4365/aa6254
Citation: https://doi.org/10.5194/egusphere-2024-301-AC4
-
AC2: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
-
RC3: 'Comment on egusphere-2024-301', Anonymous Referee #3, 17 Apr 2024
General comments
This paper describes the implementation of adaptive mesh refinement in the Vlasiator hybrid code. The authors use a series of metrics previously used in MHD codes to prescribe where refinement takes place, and perform a series of simulations, providing statistics on the computational efficiency. While the implementation is a useful contribution, and the computational sections are well-written, the paper requires revision before it can be published due to missing physics information.
Specific comments
1) Although this is primarily a numerical paper, there is a lack of physics information on the system being studied. This inculdes
- Line 52 - what is the ion kinetic scale compared to the resolution?
- What are the physical parameters of the simulation being used for the study (eg plasma and field parameters)2) Figure 1 - What are the units in the colour plot? Also, the physics appears different in the two runs (the most obvious difference is the flux rope in the tail)?
3) Line 239 - Should this be XY and XZ planes?
4) Line 245 - What do the authors mean when referring to "foreshocks"? These do not appear to be the perturbations upstream of the quasi-parallel shock, which is the usual definition.
5) Figure 5 - The bar charts in (c)-(j) have some quantities obscured due to the overlapping. I would suggest modifying the opacity or a line plot.
6) Figure 6 - It would be helpful to provide a plot of the control in addition to the existing plots.
7) In general, the authors do not show that their simulations converge. This is alluded to in point 2), where the behaviour of the magnetotail looks different in the two runs. Without this information, whether the method works cannot be judged.
8) Table 3 - I suggest providing the total run time in the table as well, since that is of practical use.
Suggestions
1) Line 179 - Please define induced refinement.
2) I would suggest using a different font/italics for dccrg
Technical corrections1) Line 138 - move (a-e) to before the quantities. E.g (a) particle density ....
2) Line 173 - his -> this
3) There are certain parts of the text where contractions are used (e.g don't). These do not fit stylistically with the rest of the paper.Citation: https://doi.org/10.5194/egusphere-2024-301-RC3 -
AC3: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
RC3:
> Line 52 - what is the ion kinetic scale compared to the resolution?
The thermal Larmor radius (~130 km) or ion inertial length (~230 km) are not resolved in these runs, where the resolution is 2000 km on the finest level. As also discussed by Ganse et al. (2023), even 1000 km currently achieved at production scale is a compromise. Performing several case studies at production run resolution would require significant computational resources through one or more dedicated tier-0 resource applications. We will include this discussion point in the paper.
> What are the physical parameters of the simulation being used for the study (eg plasma and field parameters)
Thank you for pointing out this omission. The simulation was initialised with a solar wind proton temperature of 5 * 105 K, proton number density of 1/cm^3, and a solar wind velocity of -750 km/s in the x-direction. Within 30 * 103 km from the origin, the plasma is at rest, with velocity tapering sinusoidally from a distance of 30 * 103 to a distance of 100 * 103 km, from where the full solar wind velocity is used. The background field consists of -5 nT IMF flux in the z-direction and a vector dipole field equivalent to Earth’s. We will provide the physical setup in the revision.
> Figure 1 - What are the units in the colour plot? Also, the physics appears different in the two runs (the most obvious difference is the flux rope in the tail)?
Units in Figure 1 are 1/m^3, we will add this to the revision. Physical differences stem from the lower run only having run for 50s with AMR, with kinetic effects remaining unresolved during the 500s initialization on the lower resolution. Thus, the unresolved 500 second initialization period allows global but not kinetic features to form, and the full resolution should be allowed to impact the results for several ion gyroperiods before using the data for scientific analysis.
> Line 239 - Should this be XY and XZ planes?
Yes, we will correct this in the revision.
> Line 245 - What do the authors mean when referring to "foreshocks"? These do not appear to be the perturbations upstream of the quasi-parallel shock, which is the usual definition.
We acknowledge that it is unusual to refer to the structures in Figure 6 as foreshocks. However, with the Bz IMF, the flanks do in fact build up to a quasi-parallel shock, even though the shock-normal plasma flow is miniscule. From a point of view of particle shock physics, we suggest that this region is still best described as a foreshock, even if the upstream plasma flows more in parallel with the shock front than directed towards it.
> Figure 5 - The bar charts in (c)-(j) have some quantities obscured due to the overlapping. I would suggest modifying the opacity or a line plot.
The histograms in Figure 5 (c)-(j) are stacked, i.e. the total height of the bars show the portion of spatial volume with some value of alpha, with the heights of the coloured sections of the bars showing the portion of cells on each refinement level. We apologise for the confusion and will clarify this in the revision.
> Figure 6 - It would be helpful to provide a plot of the control in addition to the existing plots
Thank you for the suggestion. Plots of the control run are given in Figure 2, and we will replicate them in Figure 6 for easier comparison.
> In general, the authors do not show that their simulations converge. This is alluded to in point 2), where the behaviour of the magnetotail looks different in the two runs. Without this information, whether the method works cannot be judged.
Thank you for raising an interesting point. The physics observed in the control and AMR runs are different, as for example the lower run in Figure 1 has only been refined in the tail for 50 seconds. Indeed the simulations haven’t converged and in an absolute sense cannot converge – different local resolutions affect physical phenomena such as reconnection rate through numerical resistivity (Rembiasz et al., 2017). It is also known that some kinetic phenomena cannot be described unless sufficient resolution is achieved (Dubart et al, 2020). Running longer with AMR enabled is expected to result in more qualitatively similar structures; the short 50 s period of refinement was meant to determine the shape of refined regions, thus ascertaining the relevance of the chosen refinement parameters.
> Table 3 - I suggest providing the total run time in the table as well, since that is of practical use.
We will provide this in Table 3.
> Line 179 - Please define induced refinement.
Thank you for the clarification request. Induced refinement is dccrg behaviour for enforcing maximum refinement differences between neighbours. Essentially, refining a cell whose neighbours are too coarse refines those neighbours. We will include this additional information in the revision.
> I would suggest using a different font/italics for dccrg
The authors acknowledge that the lowercase initialism dccrg is less than ideal for readability. We will capitalise it in the revision.
> Line 138 - move (a-e) to before the quantities. E.g (a) particle density ....
> Line 173 - his -> this
> There are certain parts of the text where contractions are used (e.g don't). These do not fit stylistically with the rest of the paper.We will make these technical corrections in the revision.
Citation: https://doi.org/10.5194/egusphere-2024-301-AC3 -
AC4: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
References:
Dubart, M., Ganse, U., Osmane, A., Johlander, A., Battarbee, M., Grandin, M., Pfau-Kempf, Y., Turc, L., & Palmroth, M. (2020). Resolution dependence of magnetosheath waves in global hybrid-Vlasov simulations. Annales Geophysicae, 38(6), 1283–1298. https://doi.org/10.5194/angeo-38-1283-2020
Ganse, U., Koskela, T., Battarbee, M., Pfau-Kempf, Y., Papadakis, K., Alho, M., Bussov, M., Cozzani, G., Dubart, M., George, H., Gordeev, E., Grandin, M., Horaites, K., Suni, J., Tarvus, V., Kebede, F. T., Turc, L., Zhou, H., & Palmroth, M. (2023). Enabling technology for global 3D + 3V hybrid-Vlasov simulations of near-Earth space. Physics of Plasmas, 30(4), 042902. https://doi.org/10.1063/5.0134387
Rembiasz, T., Obergaulinger, M., Cerdá-Durán, P., Aloy, M.-Á., & Müller, E. (2017). On the Measurements of Numerical Viscosity and Resistivity in Eulerian MHD Codes. The Astrophysical Journal Supplement Series, 230(2), 18. https://doi.org/10.3847/1538-4365/aa6254
Citation: https://doi.org/10.5194/egusphere-2024-301-AC4
-
AC3: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
-
AC1: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
We thank the referees for their comments and feedback, addressed below.
RC1:
> Line 64: Does Vlasiator use the real mass ratio?
Vlasiator uses the real mass ratio for protons and electrons since in the hybrid model electrons aren’t explicitly modelled, obviating the need for a non-physical mass ratio. Real electron masses are used in the electron pressure gradient term of Ohm’s law.
> Line 70: Please elaborate more about what spatial optimizations are needed.
The further spatial optimizations on line 70 are in reference to mesh refinement introduced in the following sub-section. We will clarify this in the revision.
> Line 84: AMR is not used in MHD-AEPIC. MHD-AEPIC is dynamically apply kinetic physics in regions that are needed. The grid resolution is not varying in space. BATS-R-US indeed uses AMR in the MHD grid. I suggest removing the MHD-AEPIC citation here.
MHD-AEPIC citation will be removed from line 84, thank you for the correction
> Figure 5(a): Why the cell number increases for the unrefined run?
In figures 5a and 5b the plotted cell count is the entire phase-space i.e. including velocity space which fills out as the state evolves. It is a measure of total computational/memory load. We will amend the figure to clarify this.
> Table 1: Please list the total runtime of each run in the table, that would justify using AMR in the future production runs, even the scaling seems to be bad.
We will provide total runtime in Table 3.
Citation: https://doi.org/10.5194/egusphere-2024-301-AC1 -
AC2: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
RC2:
> Was OpenMP used in the test runs? Regardless of the answer, could the authors provide additional discussion on its impact on performance, particularly in terms of load balancing?
OpenMP was used in all tests. Each process ran on two physical cores with hyperthreading, for a total of four OpenMP threads per process. Task balance was not the principal factor investigated, so every run used the same amount of cores per task. We will add the used threading settings in the revision.
> It is speculated that the load balance was non-ideal because the weights used in the load balancing algorithm did not account for changes in the mesh. Would more frequent load balancing help mitigate this issue?
More frequent load balancing is expected to help with AMR performance, or rather, not refining before every load balance. The intention was to “stress-test” AMR to see the overhead of refinement itself on maximal cadence in comparison to load balance. Load balancing itself also incurs a cost, so too frequent load balances will also be inefficient.
> I noticed in the Acknowledgements that the simulations were performed on CSC Mahti supercomputer. However, while reading the manuscript, I had questions about this. It would help the reader if a brief description of the computer were included alongside the presentation of the test results
Detailed information of the Mahti supercomputer is available in the Mahti documentation (https://docs.csc.fi/computing/systems-mahti/) – in these tests the CPU nodes were used. We will include a description in the revision.
> Line 138, starting with 'These gradients are': Technically speaking, an expression like (delta B)^2 / (2 mu_0) is not the gradient of the magnetic field energy, which would be B delta B / mu_0. To avoid confusion, could the authors revise this sentence?
> There is a typo in Line 173. `his` -> `this`?Thank you for the observant correction. The mention of gradients and the typo will be corrected in the revision.
Citation: https://doi.org/10.5194/egusphere-2024-301-AC2 -
AC3: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
RC3:
> Line 52 - what is the ion kinetic scale compared to the resolution?
The thermal Larmor radius (~130 km) or ion inertial length (~230 km) are not resolved in these runs, where the resolution is 2000 km on the finest level. As also discussed by Ganse et al. (2023), even 1000 km currently achieved at production scale is a compromise. Performing several case studies at production run resolution would require significant computational resources through one or more dedicated tier-0 resource applications. We will include this discussion point in the paper.
> What are the physical parameters of the simulation being used for the study (eg plasma and field parameters)
Thank you for pointing out this omission. The simulation was initialised with a solar wind proton temperature of 5 * 105 K, proton number density of 1/cm^3, and a solar wind velocity of -750 km/s in the x-direction. Within 30 * 103 km from the origin, the plasma is at rest, with velocity tapering sinusoidally from a distance of 30 * 103 to a distance of 100 * 103 km, from where the full solar wind velocity is used. The background field consists of -5 nT IMF flux in the z-direction and a vector dipole field equivalent to Earth’s. We will provide the physical setup in the revision.
> Figure 1 - What are the units in the colour plot? Also, the physics appears different in the two runs (the most obvious difference is the flux rope in the tail)?
Units in Figure 1 are 1/m^3, we will add this to the revision. Physical differences stem from the lower run only having run for 50s with AMR, with kinetic effects remaining unresolved during the 500s initialization on the lower resolution. Thus, the unresolved 500 second initialization period allows global but not kinetic features to form, and the full resolution should be allowed to impact the results for several ion gyroperiods before using the data for scientific analysis.
> Line 239 - Should this be XY and XZ planes?
Yes, we will correct this in the revision.
> Line 245 - What do the authors mean when referring to "foreshocks"? These do not appear to be the perturbations upstream of the quasi-parallel shock, which is the usual definition.
We acknowledge that it is unusual to refer to the structures in Figure 6 as foreshocks. However, with the Bz IMF, the flanks do in fact build up to a quasi-parallel shock, even though the shock-normal plasma flow is miniscule. From a point of view of particle shock physics, we suggest that this region is still best described as a foreshock, even if the upstream plasma flows more in parallel with the shock front than directed towards it.
> Figure 5 - The bar charts in (c)-(j) have some quantities obscured due to the overlapping. I would suggest modifying the opacity or a line plot.
The histograms in Figure 5 (c)-(j) are stacked, i.e. the total height of the bars show the portion of spatial volume with some value of alpha, with the heights of the coloured sections of the bars showing the portion of cells on each refinement level. We apologise for the confusion and will clarify this in the revision.
> Figure 6 - It would be helpful to provide a plot of the control in addition to the existing plots
Thank you for the suggestion. Plots of the control run are given in Figure 2, and we will replicate them in Figure 6 for easier comparison.
> In general, the authors do not show that their simulations converge. This is alluded to in point 2), where the behaviour of the magnetotail looks different in the two runs. Without this information, whether the method works cannot be judged.
Thank you for raising an interesting point. The physics observed in the control and AMR runs are different, as for example the lower run in Figure 1 has only been refined in the tail for 50 seconds. Indeed the simulations haven’t converged and in an absolute sense cannot converge – different local resolutions affect physical phenomena such as reconnection rate through numerical resistivity (Rembiasz et al., 2017). It is also known that some kinetic phenomena cannot be described unless sufficient resolution is achieved (Dubart et al, 2020). Running longer with AMR enabled is expected to result in more qualitatively similar structures; the short 50 s period of refinement was meant to determine the shape of refined regions, thus ascertaining the relevance of the chosen refinement parameters.
> Table 3 - I suggest providing the total run time in the table as well, since that is of practical use.
We will provide this in Table 3.
> Line 179 - Please define induced refinement.
Thank you for the clarification request. Induced refinement is dccrg behaviour for enforcing maximum refinement differences between neighbours. Essentially, refining a cell whose neighbours are too coarse refines those neighbours. We will include this additional information in the revision.
> I would suggest using a different font/italics for dccrg
The authors acknowledge that the lowercase initialism dccrg is less than ideal for readability. We will capitalise it in the revision.
> Line 138 - move (a-e) to before the quantities. E.g (a) particle density ....
> Line 173 - his -> this
> There are certain parts of the text where contractions are used (e.g don't). These do not fit stylistically with the rest of the paper.We will make these technical corrections in the revision.
Citation: https://doi.org/10.5194/egusphere-2024-301-AC3 -
AC4: 'Comment on egusphere-2024-301', Leo Kotipalo, 30 May 2024
References:
Dubart, M., Ganse, U., Osmane, A., Johlander, A., Battarbee, M., Grandin, M., Pfau-Kempf, Y., Turc, L., & Palmroth, M. (2020). Resolution dependence of magnetosheath waves in global hybrid-Vlasov simulations. Annales Geophysicae, 38(6), 1283–1298. https://doi.org/10.5194/angeo-38-1283-2020
Ganse, U., Koskela, T., Battarbee, M., Pfau-Kempf, Y., Papadakis, K., Alho, M., Bussov, M., Cozzani, G., Dubart, M., George, H., Gordeev, E., Grandin, M., Horaites, K., Suni, J., Tarvus, V., Kebede, F. T., Turc, L., Zhou, H., & Palmroth, M. (2023). Enabling technology for global 3D + 3V hybrid-Vlasov simulations of near-Earth space. Physics of Plasmas, 30(4), 042902. https://doi.org/10.1063/5.0134387
Rembiasz, T., Obergaulinger, M., Cerdá-Durán, P., Aloy, M.-Á., & Müller, E. (2017). On the Measurements of Numerical Viscosity and Resistivity in Eulerian MHD Codes. The Astrophysical Journal Supplement Series, 230(2), 18. https://doi.org/10.3847/1538-4365/aa6254
Citation: https://doi.org/10.5194/egusphere-2024-301-AC4
Peer review completion
Journal article(s) based on this preprint
Data sets
AMR Test Configuration Leo Kotipalo https://doi.org/10.23729/f7f9d95d-1e23-49e3-9c35-1d8e26c47bf7
Model code and software
fmihpc/vlasiator: Vlasiator 5.3 Yann Pfau-Kempf, Sebastian von Alfthan, Urs Ganse, Markus Battarbee, Leo Kotipalo, Tuomas Koskela, Ilja, Arto Sandroos, Kostis Papadakis, Markku Alho, Hongyang Zhou, Miro Palmu, Maxime Grandin, Jonas Suni, Dimitry Pokhotelov, and Konstatinos Horaites https://doi.org/10.5281/zenodo.10600112
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 310 | 62 | 37 | 409 | 21 | 18 |
- HTML: 310
- PDF: 62
- XML: 37
- Total: 409
- BibTeX: 21
- EndNote: 18
Viewed (geographical distribution)
| Country | # | Views | % |
|---|---|---|---|
| United States of America | 1 | 178 | 43 |
| Finland | 2 | 59 | 14 |
| China | 3 | 47 | 11 |
| Germany | 4 | 21 | 5 |
| France | 5 | 10 | 2 |
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
- 178
Markus Battarbee
Yann Pfau-Kempf
Minna Palmroth
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(1816 KB) - Metadata XML