the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 04 Dec 2024
A Pathways Analysis Dashboard prototype for multi-risk systems
Abstract. With accelerating climate change, impacts will compound and cascade, making them more complex to assess and manage. At the same time, tools that help decision makers choose between different management options are very limited. This study introduces a visual analytics dashboard prototype designed to support pathways analysis for multi-risk Disaster Risk Management (DRM). Developed through a systematic design approach, the dashboard employs interactive visualisations of pathways and their evaluation – including Decision Trees, Parallel Coordinates Plots, Stacked Bar Charts, Heatmaps, and Pathways Maps – to facilitate complex, multi-criteria decision-making under uncertainty. We demonstrate the utility of the dashboard through an evaluation with 54 participants at varying levels and disciplines of expertise. Depending on the expertise (non-experts, adaptation / DRM experts, pathways experts), users were able to interpret the options of the pathways, the performance of the pathways, the timing of the decisions and perform a system analysis that accounts for interactions between the sectoral DRM pathways with precision between 71 % and 80 %. Participants particularly valued the dashboard's interactivity, allowing for scenario exploration, adding additional information on demand, or offering additional clarifying data. Although the dashboard effectively supports comparative analysis of pathway options, the study highlights the need for additional guidance and onboarding resources to improve accessibility and opportunities to generalise the prototype developed to be applied in different case studies. Tested as a standalone tool, the dashboard may have additional value in participatory analysis and modelling settings. This study underscores the value of visual analytics for the DRM and Decision Making Under Deep Uncertainty (DMDU) communities, with implications for broader applications across complex and uncertain decision-making scenarios.
Competing interests: Marleen de Ruiter and Robert Šakić Trogrlić are editors of the Special Issue we are submitting this manuscript to.
Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this preprint. The responsibility to include appropriate place names lies with the authors.- Preprint
(12235 KB) - Metadata XML
-
Supplement
(1086 KB) - BibTeX
- EndNote
Status: final response (author comments only)
-
RC1: 'Comment on egusphere-2024-3655', Anonymous Referee #1, 03 Mar 2025
This paper introduces a prototype dashboard designed to support pathways analysis in multi-risk Disaster Risk Management (DRM). Prior to selecting visualization types, a literature review was conducted, with particular attention given to terminology. Additional details are provided in Annex A. The paper thoroughly explains the design process, methodology, and approach to selecting visual representations. Figure 1 and Table 1 help illustrate this approach, while Annex B provides further information on the chart type trials conducted in this study. The prototype dashboard was evaluated through a survey completed by over 50 participants. The survey questions are available in Annex C. Section 3 presents the survey results and assesses the performance of each visualization type. Lessons learned, key insights, and study limitations are discussed in Section 4, “Discussion.”
Overall Comments
- line 404 states that “this study offers a starting point to discuss and improve the toolset for policy analysis in the context of multi-risk DRM,” while line 432 transitions directly to the conclusion: “The prototype dashboard addresses a gap in DRM decision support tools by enabling multi-criteria and multi-risk analysis through interactive, user-centred design.” This reviewer believes that this point could be further expanded or discussed in greater depth.
- When printed, some figures—particularly those containing multiple sub-figures or screenshots—have text that appears too small and difficult to read. A revision of these figures is recommended.
- The URL of the dashboard is provided as www.pathways-analysis-dashboard.net (line 130). However, the URL leads to the survey version of the dashboard rather than a freely explorable version.
It would also be helpful to introduce the URL earlier in the paper or make it more clearly highlighted.
Minor comments
• Line 96: “The bold terms in Table 1 used for the description of the analysis operations are based on Brehmer and Munzner (2013) (definitions in Table A).” However, apart from the first column/row, there do not appear to be any bold terms in Table 1. Do you mean italicized terms instead?
• Line 129: Consider adding footnotes to provide context on Dash and Plotly for readers unfamiliar with these tools.
• Line 244: The word “Question” is capitalized for B6 but appears in lowercase for B3. Consider maintaining consistency.
• Line 363: One of the references appears to be missing.
• Line 443: There is an extra bracket in the middle of a reference.
Citation: https://doi.org/10.5194/egusphere-2024-3655-RC1 -
RC2: 'Comment on egusphere-2024-3655', Anonymous Referee #2, 20 Mar 2025
Thank you for inviting me to review the paper ‘A Pathways Analysis Dashboard prototype for multi-risk systems’. I have read the manuscript with great interest. The object of this study was to introduce a new dashboard that the authors state has been designed to support decision makers involved in multi-risk disaster risk management. The dashboard uses a pathway analysis approach coupled with a range of visualisations that can help users to navigate multi-risk DRM.
Overall, this is a well written paper, it reads fluently, and I have few minor suggestions around language and small corrections which I outline below in my minor comments. Please note that I have not exhaustively been through the references, so the authors may wish to do this. I did pick up on a few small issues with these.
There are however, in my opinion, some major revisions that will need to be addressed before the manuscript is suitable for publication in your journal. This work is missing some key detail with respect to who the target users were, how they were identified and how they were included in the evaluation of the final products. I hope that this can be rectified with some re-structuring and addition of information. I have outlined these suggestions in the ‘Major Comments’ section of my review.
I have detailed my comments below, I really hope that the authors find theses helpful in editing the manuscript. Please note that I am not able to edit the pdf and so have outlined my line-by-line comment below.
Major Comments:
Methods – In general, I find that the methods section is incomplete in terms of detail.
For example: In section 2.1, line 74, you mention target users. Who are these? How were they determined? Did you develop any kind of user stories to help you design the platform?
For your six semi structured interviews, who were the candidates, what sectors did they come from, how were they selected? What was their level of comfort with the data and approaches? Similarly, how did you select the candidates for your focus groups (line 75), what was their make up?
You state that potential users were based on previous studies – but I think you need more detail about who these are in this paper. As a reader coming to this work for the first time you would have no way of knowing who you were targeting.
Why did you select the Waal River as your case study? Is this because you were already working in this region, or because there were specific sectors there that you wanted to explore? You have included details of the case study in the supplementary material, but I would suggest that it would be beneficial to have at least a precis of some of this information in the main text.
In section 2.4.1, you have included information about the design process about the final visualisations in Appendix B. But again, I feel like there is some important information here that should be in the main body of the document, not buried in the Appendix. I would consider whether it was possible to move the entire context of this appendix into the main document.
Evaluation of the dashboard – Again, I find that there is some explanatory information missing here.
In section 3.1 – why are you assessing using these 4 questions? What do you mean that most of the questions were accurately answered? If you are only going to refer to 4 questions then I would suggest stating them in the text, rather than asking the reader to flip between the Appendix and the text for this key information. I think that this would make this section You say that questions were ‘less well answered for different reasons’ – what reasons, how do you know what the reasons are?
With respect to the 54 people who evaluated the dashboard. How were they selected? What sectors did they represent? On Page 13, Line 254 – the respondent states that they are ‘trying to put myself in the shoes of a farmer’ – were any of your respondents actually farmers, given the relevance of that sector to your case study?
Discussion –
In the limitations section I find the following statement surprising:
‘First, while the dashboard was designed for collaborative decision-making in a participatory modelling context, participants tested it as a standalone tool without any introductory presentation in the context of case studies. Some participants noted the need for additional context and training, indicating that such a complex topic requires more than an intuitive interface.’
Why didn’t the authors test the dashboard in the context in which it would most likely be used, that is with the accompanying contextual information? I am concerned that without this context the respondents’ responses are likely to be less useful – perhaps the authors could give their thoughts on this?
My larger concern here is, however, that in this section it seems to be revealed that all participants were researchers and not either policy makers or decision makers (Page 19, Line 350). They go on to say that ‘Given that early adopters of multi-risk DRM are often involved through research projects, this limitation may be acceptable’. This appears to be an incredibly research centric view of the world and I don’t think that this is indeed what Šakic Trogrlic et al. said in their paper. The way that this reads to me at the moment is that its acceptable that you haven’t included decision makers in the review of a tool to support decision making, because in fact researchers will be available to guide them with respect to multi-risk DRM. I suspect that this is not what you mean, I think this statement would benefit from: some clarity on what you mean by ‘early adopters’, how you see your visualisations being co-developed by policy partners and how this could feed into the implementation of multi-risk DRM, which we most definitely should not assume would be operationalised in partnership with research scientists.
Another concern here is why the testing of this visualisation tool was not designed to target the projected end users? On Page 19 you state that ‘The survey feedback emphasised the value of involving end users throughout the design process to minimise confusion and ensure that visualisations meet their intended purpose effectively’. But this contradicts the fact that you didn’t seem to involve the end users that you have targeted the product to. If you have indeed included them then this is unclear to me. I very much agree with this statement: ‘it seems vital for research communities such as multi-risk DRM or DMDU to not underestimate the value of taking the time to think about how to use visualisations and for what purpose’ but from your manuscript I don’t think that you have always clearly evidenced this approach.
I think that these ‘limitations’ call into question the overall usefulness of this visualization tool and undermine your conclusions that what you have produced is a tool that will be beneficial in the wider context of multi-risk DRM. I think that the authors need to therefore re-visit the manuscript in the context of these questions. Perhaps this is a case of clarifying the narrative and the language to more accurately frame the work that has been done? Or perhaps there are some gaps here that need to be addressed. I’m not entirely sure from that what I’ve read which of these is the case.
Minor Comments:
Abstract –
Page 1, Line 1: ‘With accelerating climate change, the impacts of hazards will compound and cascade’.
Page 1, Line 15: Remove ‘settings.
Introduction -
Page 2, Line 39: More information on the ‘four actors and two hazards’ – who were the actors, were they all from one sector? Which hazards were addressed.
Page 2 Line 53 - 55: This is more of a personal preference, but I don’t see the point of the text from ‘The paper is structured…’ onwards and would delete everything that follows this in this section. It’s a level of exposition I find unnecessary, Im reading the paper, therefore I know how its structured.
Methods -
Page 3, Line 63: Can you clarify the last sentence of this paragraph? I don’t really understand what you mean here.
Page 4, Line 79: ‘Early adopters’. I’m assuming that this represents the field in general, not your interviewees. Perhaps clarify the language between the first and second sentence of this paragraph
Page 4, Line 81: What do you mean by ‘low regret pathway’
Page 4, Line 97: Remove ‘Furthermore’.
Page 4, Line 98: Remove ‘Furthermore, we used’, replace with and.
Page 6, Line 125: Again, referring to a ‘wide range of users’, without identifying who these may be.
Page 7, Line 162: Unsure what you mean by ‘bring their individual shortlisted pathways to one table’.
Results -
Page 9: Are you missing a subheading? (3 Results)
Page 9, Line 190: What types of visual impairments are you referring to? Colour blindness?
Page 10, Line 202: Every time you quote one of your interviewees you use a ’ at the beginning of the phrase rather than a ‘ – suspect that this is a LaTeX issue, please check.
Page 10: Change figure captions from left to a) and right to b) throughout the document?
Page 12, Line 226: Can you clarify what you mean by evaluated here? As in you had a higher number of responses, or they found it easier to interpret.
Page 12, Line 232 - 33: remove ‘had a’ change particular struggle to particularly struggled.
Page 12, Line 235 - 26: replace ‘referred’ with ‘stated’ and ‘context’ with ‘contextual’.
Discussion –
Page 18, Line 351: I don’t think that this reference is in the reference list.
Page 19, Line 363: There is a bad LaTeX link here for a reference.
Page 19, Section 4.3: Do you have any comment on how you would iterate your dashboard to take on board the feedback that you have received?
Page 20, Line 397: The statement ‘This study also emphasises the value of interactive visualisations’ appears contradictory – you have said in multiple times in earlier sections that your testers did not fully utilise these features, please clarify.
Page 20, Line 410: ‘visualisation tools capable of clearly illustrating these complex interactions to help a decision maker’ you haven’t evidenced this by testing your approach with decision makers.
Page 20, Line 415: ‘we assumed decision makers would first tackle sector specific risk strategies before incorporating multi-sectoral interactions’ why? How do you evidence this? Does this genuinely reflect a ‘decision makers’ experience?
Conclusion –
Page 20, Line 427: Again, you have highlighted that participants valued the dashboards interactivity. But this seemingly contradicts the statements earlier on in the text, please consider how to evidence this more fully.
Figures –
Figure 2 is labelled a – e, whereas later figures are labelled ‘left / right’. Make labelling consistent, I think that labelling with letters is probably more appropriate.
The matrices in figures 5, 6, 7, 8, 9 aren’t intuitive to read, consider a different display? Perhaps bar charts?
Figure 4 – Check the x axis, category labels are overlapping. Legend boxes have been placed over data points – These need to be adjusted so that all the data points are clearly visible.
Please note I have not completed a line by line check of the Appendices or references for expediency, so authors may wish to do this a final time.
Citation: https://doi.org/10.5194/egusphere-2024-3655-RC2
Model code and software
Code to create Pathways Analysis Dashboard Infrastructure and Visualizations J. Schlumberger https://github.com/JuliusSchlumberger/Pathways_Analysis_Dashboard
Interactive computing environment
Pathways Analysis Dashboard J. Schlumberger et al. https://www.pathways-analysis-dashboard.net/
Viewed
| HTML | XML | Total | Supplement | BibTeX | EndNote | |
|---|---|---|---|---|---|---|
| 216 | 71 | 6 | 293 | 44 | 8 | 7 |
- HTML: 216
- PDF: 71
- XML: 6
- Total: 293
- Supplement: 44
- BibTeX: 8
- EndNote: 7
Viewed (geographical distribution)
| Country | # | Views | % |
|---|---|---|---|
| United States of America | 1 | 101 | 36 |
| Netherlands | 2 | 19 | 6 |
| Germany | 3 | 16 | 5 |
| France | 4 | 16 | 5 |
| United Kingdom | 5 | 13 | 4 |
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
- 101