the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 13 Sep 2023
Brief communication: lessons learnt and experienced gained from building up a global survey on societal resilience to droughts
Abstract. Drought resilience indexes are essential tools for an evidence-based decision-making process. They can be constructed by enquiring experts about the most relevant indicators. This communication presents the challenges found a priori of creating a global survey with experts in drought. The lessons learned include: (1) the heterogeneity of the conceptual background should be minimized prior the construction of the survey; (2) large number of indicators decreases the engagement of respondents; (3) the format should balance response rate and accuracy, (4) raising diverse experts by knowledge areas, experience and regional representations is difficult, but crucial to minimize biased results.
-
Notice on discussion status
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
-
Preprint
(621 KB)
-
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(621 KB) - Metadata XML
- BibTeX
- EndNote
- Final revised paper
Journal article(s) based on this preprint
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2023-2042', Anonymous Referee #1, 20 Sep 2023
The paper offers a fascinating insight into the challenges of creating a global survey, even when this is aimed at experts. It is great to see researchers sharing their experiences, tips, and tricks; it is beneficial for the community when facing similar challenges. Nevertheless, the paper suffers from major limitations, both conceptually and structurally.
The paper presents itself as conceptually confused. First of all, the central topic of the survey, resilience, is poorly discussed. This is particularly crucial since the term is famously contentious, and slippery, and has been fought over by several disciplines. Instead, the researchers choose to define it using the Sendai framework's definition, which is itself vague and meant for high-level decision-makers. This definition comes from the UNISDR terminology dictionary (which is the collection of working definitions used by UNDRR), dating back to 2009. Since then, resilience has largely evolved as a term and has become the focus of much scholarly debate, itself unaddressed in the paper. The authors claim a rigid application of the framework, yet this is seemingly in contrast with the chosen framework and with what they present. This brings me to my next point, the lack of clarity on what resilience is is reflected in the indicators chosen for the index. By their own admission, the authors followed a vulnerability, hazard, and exposure framework, which is usually used to define risk. Hence, the author's definition of resilience is seemingly the inverse of risk. If this is the case, it should be discussed, as it can be the cause of the misunderstandings and resistance that the authors encountered when initially sharing their survey with the respondents. The authors fail to acknowledge how epistemological differences across disciplines can be at the root of the issue of defining resilience, instead indicating “significant variation between regions” (line 147) as the issue. Instead, the usefulness of resilience indicators and how they are seemingly hard to make is repeated several times without providing references.
Structurally the paper is uneven. First of all, the introduction gives an unclear idea of what the scope of the paper is. If your intention is to talk about the challenges and limitations of global surveys you should do that from the start. As it stands, the introduction talks about the importance of global resilience indexes. I suggest focussing from the get-go on the importance of global surveys, their challenges, and what the paper offers. Why resilience indexes are important is secondary. The fact that the authors created a resilience index should be addressed as long as it is used to show the challenges of creating a global index (the challenges of working with a slippery concept, for example), while the importance of a global resilience index is secondary. This can be addressed in two sentences with a good amount of references to support it. I believe that changing the focus of the introduction can also help improve the quality of the paper conceptually, by shifting the focus away from resilience and towards the survey itself. Write the whole introduction about global surveys and global indexes.
Finally, the authors make little use of references, especially in critical statements, such as the use of resilience indexes, and the presence of such indexes in the literature. Additionally, the terminology is often confusing and unclear. While the aim of the study is that of creating a resilience index, the authors refer to it as risk (lines 112, 146, 166), risk and resilience (lines 113, 136, 155). Are the two terms related in your analysis? If yes, state it clearly. Additionally, the paper opens with the statement “Drought is an omnipresent natural disaster”. The term natural disaster is generally considered very controversial (despite being used by the IPCC, for example) and might alienate readers with a background in DRR; I suggest disaster or natural hazard.
Overall, the paper contains very useful information on how to run a global survey of experts that I am grateful the authors decided to share with the community. I particularly appreciated the author’s commitment to highlighting the often-unaddressed a priori assumptions that go into making a questionnaire. Additionally, I found extremely useful the tips and tricks they shared, especially when specific (such as not having more than 40 questions, or keeping it within 15 minutes). I believe that by addressing the structural unevenness of the paper and reducing the focus on the resilience aspect itself, the authors will be able to create a piece of research worth sharing with the community. Thank you very much for your contributions.
Thank you!
-
AC1: 'Reply on RC1', Marina Batalini de Macedo, 07 Dec 2023
1) The authors would like to thank the comments from RC1, as they contribute to improving the quality of the text and how to present the main questions and contributions of this brief communication. Our motivation in writing this brief communication was the lack of papers or materials discussing the challenges of building global surveys, especially in tricky fields where we have conceptual divergences - such as resilience.
It is important to highlight that our research project has the main purpose of creating a decision support system (DSS) to help decision-makers evaluate the impact of different actions on decreasing vulnerability and improving the resilience of local farm communities to agricultural droughts. Creating a resilience composite index, with global representativeness, is one tool for our DSS. In this context, it was necessary to have input and perspectives from different experts across the globe, with different backgrounds, therefore we chose to do a global survey on drought indicators.
Even though this wasn’t the main part and contribution of our research project, this phase was the most time consuming and critical of our research. We spent approximately one year constructing the survey, due to the challenges presented in this brief communication. We believe that it is important to discuss the process of building the survey to prevent other researchers from passing the same problems that we had and improving the use and interpretation of this important tool. The importance of preparing the survey itself was also discussed by Elangovan and Sundaravel (2021) for any generic field. Here, we would like to complement the challenges in the resilience field.
References:
Elangovan and Sundaravel (2021). Method of preparing a document for survey instrument validation by experts. Available at: https://www.sciencedirect.com/science/article/pii/S2215016121001199
2) Regarding the comment related to the conceptual confusion regarding resilience, the authors agree with the RC1 and would like to have the opportunity to improve the text discussion on that. Resilience is indeed a contentious and slippery term, with different definitions for its quantification and components.This was actually our first main challenge, as described in lines 84 - 95. To create a global survey, replicable and interpretable, it was necessary to have a good definition of the resilience concept that we were using. The different concepts make it difficult to categorize the indicators in the resilience components and may affect the perception of their relevance to the respondent.
Therefore, we have decided to adopt the resilience term from a disaster risk reduction approach, as presented in the Sendai framework. We understand that this approach is meant for high-level decision-makers, but since the final aim of our research project is to create a decision support system for different international contexts, this definition was the most suitable for us. From the disaster risk reduction approach, it is important first to understand the evolution of how to deal with disasters, from an international and decision-making perspective.
Past discussions attempting to reduce the impacts of disasters on the community had a focus on disaster management, defined by the UNDRR (2021) as "the organization, planning and application of measures preparing for, responding to and recovering from disaster”. Therefore, the disaster management does not necessarily aim at averting or eliminating the threats, but decreasing the negative impacts resulting from the event and recovering as fast as possible to the original (or better) state of the system. Recently, new international discussions and documents on this topic have shifted from the approach of disaster management to disaster risk management. Disaster risk management is defined as “the application of disaster risk reduction policies and strategies to prevent new disaster risk, reduce existing disaster risk and manage residual risk, contributing to the strengthening of resilience and reduction of disaster losses” (UNDRR, 2021) aiming at actions in different timescales and with a final focus on increasing the economic, social, health and environmental resilience.
This change of approach in dealing with and managing disasters has been incorporated in the Sendai Framework for Disaster Risk Reduction 2015-2030 report (UNDRR, 2015). The report presents the importance of pre-disaster actions, such as prevention, mitigation, development and implementation of appropriate actions focused on preparing and effectively responding to disasters. To this end, it is emphasized the importance of risk assessment actions and dissemination of location-based disaster risk information, for a science-policy interface in the decision-making process in disaster risk management (called risk-informed-decision-making).
As previously mentioned, the ultimate goal of disaster risk management is to increase and strengthen resilience in its different aspects, with a greater focus on people's health and livelihood. The UNDDR (2021) defines resilience as “the ability of a system or community to anticipate, resist, prepare, respond to and recover from an event with multiple risks, with the least possible harm to social, economic, and environmental well-being”. For disaster risk reduction actions, it is important to establish concrete targets and to quantify the resilience from indicators and in different timeframes (UNDRR, 2015) for its components of speed, robustness, resourcefulness, and redundancy (Karamouz et al., 2016). Several indices have been proposed over the years to represent the level of resilience of a given system to a disruptive event. In general, the resilience assessment requires the identification of the risks present in the system due to the disruptive event and the risk management policy adopted by local agents to prevent its occurrence or reduce its impacts along the system's chain.
The risk analysis can be represented by a function that correlates the probability of occurrence of the disruptive event (H), the vulnerability of the different components of the studied system (V) and the exposure to risk of the different components (E), so that vulnerability and exposure quantitatively represent the potential impacts of the event on the system (Merz et al., 2014). Within the disaster risk management and risk-oriented-decision-making approach, as adopted in the Sendai report, the risk analysis stage is of fundamental importance and a precursor to the decision-making process.
To quantify the actions proposed during the risk management stage, it is important to evaluate the type of the proposed action (structural/measures and non-structural/instruments – Schanze, 2006), its temporal component (anticipation, prevention, mitigation, preparedness, response, recovery – UNDRR, 2015; Toklu, 2017), the magnitude of the impacts if the proposed action fails (high, medium and low). According to these components, the actions can be correlated with the different system capacities that help reduce the disaster risk and further impacts, therefore, improving resilience, such as adaptive capacity, coping capacity, and transformative capacity.
From our literature review, we have identified that indicators related to quantifying the hazard component of the agricultural drought risk were already well established. Therefore, our focus on this survey was to advance on identifying indicators related to the risk impacts (vulnerability and exposure) and the system's capacities to deal with the risk and increase resilience (adaptive, coping and transformative capacity).
We agree with the RC1 that all of this discussion on the conceptual framework and terminology should be incorporated into the manuscript since in its current form it is confusing. The discussion about the resilience framework was one of the most important parts of our research, which ended by choosing to adopt the resilience concepts from a disaster risk reduction approach, due to its use of the Sendai framework. We failed to synthesize this discussion in a clear format in the text, and we hope we can have the opportunity to make these adjustments, such as changing the term natural disaster only to disaster.
Finally, the potential ambiguity that resilience can bring to the respondents, the method adopted is as heuristically addressed as coevolutionary-conceived for unprecedented climate-driven extremes. Hence, our method can be adapted for the intercomparison of different case studies (i.e. Kreibich et al, 2022) and for sharing novel metrics of open-access datasets under F.A.I.R. principles (e.g. Kreibich et al, 2023).
References:
Karamouz, M., Rasoulnia, E., Zahmatkesh, Z., Olyaei, M. A., & Baghvand, A. (2016). Uncertainty-based flood resiliency evaluation of wastewater treatment plants. Journal of Hydroinformatics, 18(6), 990-1006.
Kreibich, H., et al. (2022) The challenge of unprecedented floods and droughts in risk management. Nature 608, 80–86 https://doi.org/10.1038/s41586-022-04917-5
Kreibich, H., et al (2023) Panta Rhei benchmark dataset: socio-hydrological data of paired events of floods and droughts, Earth System Science Data 15(5) https://doi.org/10.5194/essd-15-2009-2023
Merz, B., Hall, J., Disse, M., & Schumann, A. (2010). Fluvial flood risk management in a changing world. Natural Hazards and Earth System Sciences, 10(3), 509-527.
Toklu, A. (2017). Improving Organisational Performance with Balanced Scorecard in Humanitarian Logistics: A Proposal for Key Performance Indicators. International Journal of Academic Research in Accounting, Finance and Management Sciences, 7(1), 131-137.
UNDRR – United Nations Office for Disaster Risk Reduction. (2015). Sendai Framework for Disaster Risk Reduction 2015-2030.
UNDRR – United Nations Office for Disaster Risk Reduction. (2021). Terminology. Available in: < https://www.undrr.org/terminology>. Access on: Jun 2021.
3) As for the introduction part, we agree with RC1 that the focus should change from the importance of the global resilience index to a more focus on the construction of the survey itself. As discussed before, our research project's purpose is on using a global drought index as a tool for DSS and this led us to emphasize the importance of the index in the introduction. However, we agree with the RC1 that this index is not the focus of this brief communication. Therefore, in a revised version of the manuscript we will restructure the introduction to a more on point on the survey itself. Additionally, we agree with the reviewer that the term natural disaster may give a misunderstanding of the socio-natural-antropic mix that composes the disasters. Therefore, in the revised version we will change the term natural disaster only for disaster.4) Concerning the little use of references, we are sorry that we didn´t make more use of references in critical statements, but we should pay attention to the limitation of a maximum 20 references on brief communications. We believe it is important to prioritize the references used for each indicator selected on Table 1.
Despite that, we would like to provide here examples of the literature on resilience indices and global index that were part of our literature review to construct the survey and to support the use of resilience indexes, which can be included in the revised version of the brief communication.
One of the main literature on indices and indicators is provided by the Integrated Drought Management Programme (IDMP) (WMO and GWP, 2016), presenting the definition of drought impact as “an observable loss or change at a specific time because of drought”. In the handbook provided by them, they discuss the importance of using index/indices to help monitor, early warning and support decision-making for a risk-based drought management policy. Since drought is a many-sided disaster, using composite or hybrid indicators may be more representative. Additionally, this handbook provides guidelines for selecting indicators.
WMO and GWP (2016). Handbook of Drought Indicators and Indices. Available at: https://www.droughtmanagement.info/literature/GWP_Handbook_of_Drought_Indicators_and_Indices_2016.pdf
Other literature used in this study are provided bellow:
Alonso, Catarina, Celia M Gouveia, Ana Russo, and Patrícia Páscoa. 2019. "Crops' exposure, sensitivity and adaptive capacity to drought occurrence." Natural Hazards and Earth System Sciences 19 (12):2727-43.
Antwi-Agyei, Philip, Evan D. G. Fraser, Andrew J. Dougill, Lindsay C. Stringer, and Elisabeth Simelton. 2012. "Mapping the vulnerability of crop production to drought in Ghana using rainfall, yield and socioeconomic data." Applied Geography 32 (2):324-34.
Bachmair, Sophie, Kerstin Stahl, Kevin Collins, Jamie Hannaford, Mike Acreman, Mark Svoboda, Cody Knutson, Kelly Helm Smith, Nicole Wall, and Brian Fuchs. 2016. "Drought indicators revisited: the need for a wider consideration of environment and society." Wiley Interdisciplinary Reviews: Water 3 (4):516-36.
Blauhut, Veit. 2020. "The triple complexity of drought risk analysis and its visualisation via mapping: a review across scales and sectors." Earth-Science Reviews 210:103345.
Dabanli, Ismail. 2018. "Drought hazard, vulnerability, and risk assessment in Turkey." Arabian Journal of Geosciences 11 (18):1-12.
Dardonville, Manon, Christian Bockstaller, and Olivier Therond. 2021. "Review of quantitative evaluations of the resilience, vulnerability, robustness and adaptive capacity of temperate agricultural systems." Journal of Cleaner Production 286:125456.
Elagib, Nadir Ahmed. 2014. "Development and application of a drought risk index for food crop yield in Eastern Sahel." Ecological Indicators 43:114-25.
Fang, Dan, Zhuowei Hu, Zhiheng Wang, and Guangyao Duan. 2011. Drought disaster risk assessment and mapping at different scales based on RS and GIS. Paper presented at the 2011 19th International Conference on Geoinformatics.
Jayanthi, Harikishan, and Greg Husak. 2013. "A probabilistic approach to assess agricultural drought risk." Background Paper Prepared for the Global Assessment Report on Disaster Risk Reduction.
Murthy, C. S., Manoj Yadav, J. Mohammed Ahamed, B. Laxman, R. Prawasi, M. V. R. Sesha Sai, and R. S. Hooda. 2015. "A study on agricultural drought vulnerability at disaggregated level in a highly irrigated and intensely cropped state of India." Environmental Monitoring and Assessment 187 (3):140.
Xu, Huafeng, Kexin Xu, and Yingjie Yang. 2021. "Risk assessment model of agricultural drought disaster based on grey matter-element analysis theory." Natural hazards 107 (3):2693-707.
4) We appreciate the comments made by RC1 and we hope we have addressed them all here in the reply and we hope to have the opportunity to make the corrections to the manuscript, to improve the final text of the brief communication.
Citation: https://doi.org/10.5194/egusphere-2023-2042-AC1
-
AC1: 'Reply on RC1', Marina Batalini de Macedo, 07 Dec 2023
-
RC2: 'Comment on egusphere-2023-2042', Anonymous Referee #2, 09 Oct 2023
Dear authors
I respect the publication of challenging research and unsuccessful experiments. I agree that it is important to reflect on and write about so other researchers can overcome hurdles and avoid the same problems.However, the manuscript as is shows a lack of theoretical grounding in the survey setup and also contains quite some spelling mistakes and sloppiness. The latter can be solved easily, but I am not sure the former can be improved easily as the manuscript reports about a past stakeholder engagement.
About the presentation: the title contains two language errors: lessons learned and experiences. In the abstract, the plural of index should be indices and raising should be reaching, I presume.
Line 30+ is quite vague (I do not understand what is meant there) and in general this alinea misses quite some references in my opinion.
The introduction does not base the presented research well in existing literature. Significant studies on drought vulnerability including stakeholder engagement like Meza et al. Ahmadalipour et al. should have been explained and reflected on, showing how the presented study builds on it. (see also: European Commission, Joint Research Centre, Hagenlocher, M., Vogt, J., Meza, I. et al., Drought vulnerability indicators for global-scale drought risk assessments – Global expert survey results report, Publications Office, 2019, https://data.europa.eu/doi/10.2760/73844)
Line 41 misses a word "to"
I like the short description of multiple strategies to collect expert data but I would have loved to see references to authors who used these approaches in contexts relatable to the one described in the manuscript. Especially the followed Delphi technique should have been more elaborated and with references to relevant works.
Content wise, a better embedding in existing studies would have helped designing the study. For example, the use of the latest UNDRR terminology or the IPCC glossary on terms such as vulnerability sensitivity coping capacity and resilience would have helped starting discussions and creating a common understanding with the stakeholders. Now, the three components of risk (as also acknowledged in the Sendai Framework), hazard, exposure, vulnerability, are confused and mixed up both in the manuscript text and I fear also in the study. (e.g line 112)
I like the narrow focus for this study as in "the risk for whom / what" being small farms - I would have loved to see the full list of 136 indicators in supplementary (it is also unclear how this list was extracted and why it was not based on a structured literature review).
It remains confusing to me as to why indicators related to hazard are part of the resilience framework.
I appreciate that the authors realised that co-dependent variables would hamper the study and that too much indicators (or indices?) would not allow for a good survey. However, the next steps are not very transparent: How is the pre-selected list created (how many indicators are left?)? How are ratings discussed in the group discussions (and who was present there?) Why are relevancy, ease of understanding, accessibility, objectivity and consistency chosen as evaluation metrices (subjective choice? literature behind it?)?
Also here, references to show on which survey design choices are based, are lacking.
I am inspired by your approach to use publications in the web of science and scopus to reach scientists, but would have hoped that the authors had used existing policy networks around drought (IDMP, UNCCD, WMO, FAO) to reach other types of stakeholders (the do have databases of practitioners....).
I know getting responses is a real challenges and 120 is not too bad, but indeed the distribution in terms of sector, background and region is quite relevant. However, I would have loved to see some of the results / distribution analysis of the 120 respondents and also about the discussion group to understand exactly who was reached.
It would be good to disclose the survey in supplementary, as now it remains unclear whether for example open questions were asked to check for missing indices.
In general, there were a few critical steps from standard "how to built stakeholder surveys - literature" that were not followed (For example, I do not read anything about a piloting phase) hence the - to the authors - disappointing or challenging result.
Therefore, I feel this is a very interesting learning experience that mainly points to the relevance of using other studies, guidance frameworks, textbooks on qualitative research methods, when building a global survey.
However, I am not sure it is suitable in its current form for publication in NHESS.Citation: https://doi.org/10.5194/egusphere-2023-2042-RC2 -
AC2: 'Reply on RC2', Marina Batalini de Macedo, 07 Dec 2023
1) The authors would like to thank the comments from RC2, as it contributes to improving the quality of the text and how to present the main questions and contributions of this brief communication. Our motivation in writing this brief communication was the lack of papers or materials discussing the challenges of building global surveys, especially in tricky fields where we have conceptual divergences - such as resilience.
It is important to highlight that our research project has the main purpose of creating a decision support system (DSS) to help decision-makers evaluate the impact of different actions on decreasing vulnerability and improving the resilience of local farm communities to agricultural droughts. Creating a resilience composite index, with global representativeness, is one tool for our DSS. In this context, it was necessary to have input and perspective from different experts across the globe, with different backgrounds, therefore we chose to do a global survey on drought indicators.
Even though this wasn’t the main part and contribution of our research project, this phase was the most time consuming and critical of our research. We spent approximately one year constructing the survey, due to the challenges presented in this brief communication.
We believe that addressing the difficulties of the survey construction process is not the focus of the discussion of the papers for fear of the researchers, by sharing the problems obtained, weakening confidence in their final results presented later. However, we need to face this fear, as sharing challenges is also an important gain of knowledge. This was an issue found by other authors as well, e.g. Baker et al. (2014) states that “..While there is a rich literature on expert elicitation approaches and protocols, there is less information available on the specifics of how an elicitation is carried out".
Reference:
Baker et al. (2014). Facing the Experts: Survey Mode and Expert Elicitation. Available at: https://www.jstor.org/stable/pdf/resrep01013.pdf
2) The authors appreciate the RC2 for paying our attention to the English language errors. We appreciate your understanding that English is not the first language of the majority of the co-authors and sometimes this can lead to some errors, that we are glad it was pointed out before the final version of the brief communication. We will take a meticulous language review of the text.
3) The reviewer commented about important references that were not considered in our manuscript, like Meza et al. Ahmadalipour et al.. We would like to highlight that Meza et al. was used as a reference and we would like to have the opportunity to also incorporate the one by Ahmadalipour et al.
We would like to highlight that we are sorry that we didn´t make more use of references in critical statements, but we should pay attention to the limitation of a maximum 20 references on brief communications. We believe it is important to prioritize the references used for each indicator selected on Table 1.
Despite that, we would like to provide here examples of the literature on resilience indices and global index that were part of our literature review to construct the survey and to support the use of resilience indexes, which can be included in the revised version of the brief communication.
One of the main literature on indices and indicators is provided by the Integrated Drought Management Programme (IDMP) (WMO and GWP, 2016), presenting the definition of drought impact as “an observable loss or change at a specific time because of drought”. In the handbook provided by them, they discuss the importance of using index/indices to help monitor, early warning and support decision-making for a risk-based drought management policy. Since drought is a many-sided disaster, using composite or hybrid indicators may be more representative. Additionally, this handbook provides guidelines for selecting indicators.
WMO and GWP (2016). Handbook of Drought Indicators and Indices. Available at: https://www.droughtmanagement.info/literature/GWP_Handbook_of_Drought_Indicators_and_Indices_2016.pdf
Other literature used in this study are provided bellow:
Alonso, Catarina, Celia M Gouveia, Ana Russo, and Patrícia Páscoa. 2019. "Crops' exposure, sensitivity and adaptive capacity to drought occurrence." Natural Hazards and Earth System Sciences 19 (12):2727-43.
Antwi-Agyei, Philip, Evan D. G. Fraser, Andrew J. Dougill, Lindsay C. Stringer, and Elisabeth Simelton. 2012. "Mapping the vulnerability of crop production to drought in Ghana using rainfall, yield and socioeconomic data." Applied Geography 32 (2):324-34.
Bachmair, Sophie, Kerstin Stahl, Kevin Collins, Jamie Hannaford, Mike Acreman, Mark Svoboda, Cody Knutson, Kelly Helm Smith, Nicole Wall, and Brian Fuchs. 2016. "Drought indicators revisited: the need for a wider consideration of environment and society." Wiley Interdisciplinary Reviews: Water 3 (4):516-36.
Blauhut, Veit. 2020. "The triple complexity of drought risk analysis and its visualisation via mapping: a review across scales and sectors." Earth-Science Reviews 210:103345.
Dabanli, Ismail. 2018. "Drought hazard, vulnerability, and risk assessment in Turkey." Arabian Journal of Geosciences 11 (18):1-12.
Dardonville, Manon, Christian Bockstaller, and Olivier Therond. 2021. "Review of quantitative evaluations of the resilience, vulnerability, robustness and adaptive capacity of temperate agricultural systems." Journal of Cleaner Production 286:125456.
Elagib, Nadir Ahmed. 2014. "Development and application of a drought risk index for food crop yield in Eastern Sahel." Ecological Indicators 43:114-25.
Fang, Dan, Zhuowei Hu, Zhiheng Wang, and Guangyao Duan. 2011. Drought disaster risk assessment and mapping at different scales based on RS and GIS. Paper presented at the 2011 19th International Conference on Geoinformatics.
Jayanthi, Harikishan, and Greg Husak. 2013. "A probabilistic approach to assess agricultural drought risk." Background Paper Prepared for the Global Assessment Report on Disaster Risk Reduction.
Murthy, C. S., Manoj Yadav, J. Mohammed Ahamed, B. Laxman, R. Prawasi, M. V. R. Sesha Sai, and R. S. Hooda. 2015. "A study on agricultural drought vulnerability at disaggregated level in a highly irrigated and intensely cropped state of India." Environmental Monitoring and Assessment 187 (3):140.
Xu, Huafeng, Kexin Xu, and Yingjie Yang. 2021. "Risk assessment model of agricultural drought disaster based on grey matter-element analysis theory." Natural hazards 107 (3):2693-707.
4) Concerning the suggestion of RC2 on the Delphi technique and more elaboration with references to relevant work, we would like to justify this because the manuscript type chosen by the group to present the challenges faced by us on constructing the global survey is the brief communications. Unfortunately, this type of manuscript does not allow more than 20 references (https://www.natural-hazards-and-earth-system-sciences.net/about/manuscript_types.html), which limited our possibilities in including on the manuscript authors that used these approaches in the drought resilience context. We have prioritized the references related to the indicators/indices selected by the authors to be part of the survey (presented in Table 1).
Despite this limitation, we would like to present here some examples of papers that used these different approaches:
Proposal for a Tool for Assessing Access to Water in Rural Communities: a Case Study in the Brazilian Semi-arid. https://link.springer.com/article/10.1007/s00267-022-01600-3
Select indicators and prioritize solutions for desertification and drought in Binh Thuan, Vietnam. https://www.tandfonline.com/doi/abs/10.1080/10042857.2016.1177315
5) The resilience framework chosen for us on this research project was based on the disaster risk reduction approach, as presented in the Sendai framework. From the disaster risk reduction approach, it is important first to understand the evolution of how to deal with disasters, from an international and decision-making perspective.
Disaster risk management is defined as “the application of disaster risk reduction policies and strategies to prevent new disaster risk, reduce existing disaster risk and manage residual risk, contributing to the strengthening of resilience and reduction of disaster losses” (UNDRR, 2021) aiming at actions in different timescales and with a final focus on increasing the economic, social, health and environmental resilience.
This approach in dealing with and managing disasters has been incorporated in the Sendai Framework for Disaster Risk Reduction 2015-2030 report (UNDRR, 2015). The report presents the importance of pre-disaster actions, such as prevention, mitigation, development and implementation of appropriate actions focused on preparing and effectively responding to disasters. To this end, it is emphasized the importance of risk assessment actions and dissemination of location-based disaster risk information, for a science-policy interface in the decision-making process in disaster risk management (called risk-informed-decision-making).
As previously mentioned, the ultimate goal of disaster risk management is to increase and strengthen resilience in its different aspects, with a greater focus on people's health and livelihood. The UNDDR (2021) defines resilience as “the ability of a system or community to anticipate, resist, prepare, respond to and recover from an event with multiple risks, with the least possible harm to social, economic, and environmental well-being”. For disaster risk reduction actions, it is important to establish concrete targets and to quantify the resilience from indicators and in different timeframes (UNDRR, 2015) for its components of speed, robustness, resourcefulness, and redundancy (Karamouz et al., 2016). Several indices have been proposed over the years to represent the level of resilience of a given system to a disruptive event. In general, the resilience assessment requires the identification of the risks present in the system due to the disruptive event and the risk management policy adopted by local agents to prevent its occurrence or reduce its impacts along the system's chain.
The risk analysis can be represented by a function that correlates the probability of occurrence of the disruptive event (H), the vulnerability of the different components of the studied system (V) and the exposure to risk of the different components (E), so that vulnerability and exposure quantitatively represent the potential impacts of the event on the system (Merz et al., 2014). Within the disaster risk management and risk-oriented-decision-making approach, as adopted in the Sendai report, the risk analysis stage is of fundamental importance and a precursor to the decision-making process.
From our literature review, we have identified that indicators related to quantifying the hazard component of the agricultural drought risk were already well established. Therefore, our focus on this survey was to advance on identifying indicators related to the risk impacts (vulnerability and exposure) and the system's capacities to deal with the risk and increase resilience (adaptive, coping and transformative capacity).
References:
Karamouz, M., Rasoulnia, E., Zahmatkesh, Z., Olyaei, M. A., & Baghvand, A. (2016). Uncertainty-based flood resiliency evaluation of wastewater treatment plants. Journal of Hydroinformatics, 18(6), 990-1006.
Merz, B., Hall, J., Disse, M., & Schumann, A. (2010). Fluvial flood risk management in a changing world. Natural Hazards and Earth System Sciences, 10(3), 509-527.
UNDRR – United Nations Office for Disaster Risk Reduction. (2015). Sendai Framework for Disaster Risk Reduction 2015-2030.
UNDRR – United Nations Office for Disaster Risk Reduction. (2021). Terminology. Available in: < https://www.undrr.org/terminology>. Access on: Jun 2021.
6) Regarding the list of the 136 indicators, the initial selection of these indicators was done from a structured literature review, as mentioned in lines 103 and 104. I am sorry if this information wasn’t clear in the text. In the revised version we can make it clearer. We appreciate the comment for displaying the full list of 136 indicators in supplementary, and we already provide them here in the attached material.
Further, the selection of the final list of indicators was made through three steps, related to the issues presented in lines 103 - 115. The first step was removing hazard indicators. In this step, 31 indicators were removed. In the second step, we removed co-dependent indicators/indices from the list, remaining the ones with more availability and easy to get. E.g. from Gini index and poverty rate, we remain with the poverty rate, since it is a more direct measurement and easier to get in different contexts. This process of eliminating co-dependent indicators/indices was made interactively in group discussion sessions. 28 indicators were removed in this process. The third step was reducing the total number of indicators/indices to avoid the survey becoming too extensive and exhaustive to answer. From this part, each participant of the group (in total 10 participants, distributed from Brazil, UK, and USA, but with different backgrounds on their birth countries and graduation courses) independently has evaluated the relevance of the indicators/indices, through a form available only for the group. From the answers in that form, in a group discussion session we have selected the 33 indicators with the higher average rating. 3 indicators were later added after the first pilot run, as they were suggested by experts on this pilot.
Additionally, during the discussion sessions with the relevance data, one of the concerns was that some indicators/indices are very interesting and relevant, but they are not easy to obtain. Therefore, it was also interesting to consider other qualitative aspects when selecting the final indicators and weights for the composite drought index, such as ease of understanding, accessibility, and objectivity. The choice of these specific metrics came from the literature (Sweya et al., 2021). The paper presented 5 complementary attributes for social resilience indicators of water supply systems (which are: affordability, availability, reliability, simplicity, and transparency). As a result, they have obtained that data availability, reliability and affordability were the most limiting factors for selecting the indicators in Tanzania. In this sense, and with a focus on the Global South, the group selected the three me metrics before mentioned to be complementary to the relevancy and adapted from Sweya et al. (2021), where understanding was used to represent the simplicity, accessibility was used as a single attribute to account for affordability and availability and objectivity was an additional attribute that we chosen to evaluate how objective is the final measure (since come of our social indicators are political measurements).
Reference:
Sweya et al. (2021). A social resilience measure tool for Tanzania’s water supply systems. Available at: https://www.sciencedirect.com/science/article/pii/S2212420921005197
7) Concerning the survey design, this was made based on guidelines for operationalizing the Delphi method (Hasson et al., 2000) and the suggestions made by Elangovan and Sundaravel (2021) on preparing documents for survey instrument validation by experts. These authors state that: “Even though there are many literatures available on the reliability and validity procedures, many researches struggle to operationalize the process.”, therefore providing a template to validate the survey instrument. However, they present a generic form of document, in which we still experienced difficulties related to the resilience field study.
Therefore, additionally, we have improved our survey design based on the evaluation of different literature that used Delphi method to access resilience indicators (e.g. Alshehri et al., 2015; Ogah et al., 2021), identifying the main problems obtained, literature on composite index for resilience (Sweya et al.; 2021), in which we could identify important complementary questions about attributes that usually are not asked in the surveys and all the relevant data are assumed to be equally easy to obtain and understand.
During this one year of constructing the survey, different survey design were tested within the group and they were being adapted based on the user experience.
We are sorry that we didn’t explain this in the text and we hope it is clear now. We would like to have the opportunity to incorporate a discussion specifically on the design on the revised version of the manuscript.
References:
Hasson et al. (2000). Research guidelines for the Delphi survey technique. Available at: https://pubmed.ncbi.nlm.nih.gov/11095242/
Elangovan and Sundaravel (2021). Method of preparing a document for survey instrument validation by experts. Available at: https://www.sciencedirect.com/science/article/pii/S2215016121001199
Alsherhri et al. (2015). Delphi-based consensus study into a framework of community resilience to disaster. Available at: https://link.springer.com/article/10.1007/s11069-014-1423-x
Ogah et al. (2021). Operationalising Community Resilience to Climate Change in Developing Countries: A Grounded Delphi Method (GDM) Approach. Available at: https://assets.researchsquare.com/files/rs-844800/v1/a597c71b-b9e8-4f50-a1b5-f7bcc3a645fd.pdf?c=1643475669
Sweya et al. (2021). A social resilience measure tool for Tanzania’s water supply systems. Available at: https://www.sciencedirect.com/science/article/pii/S2212420921005197
8) As for the suggestion of the RC2 on searching other types of stakeholders within the databases of practitioners, we appreciate this suggestion very much. We will try to contact these policy networks to have access to the database and maybe do a second round of the survey. We will also incorporate this suggestion in the revised version of the manuscript.
9) Regarding the comment of RC2 on presenting results on distribution analysis on the 120 respondents, the focus of this brief communication was to present the main challenges on constructing a global survey a priori. The results obtained by the survey, such as the distribution of the 120 respondents, are presented in an additional paper (Sass et al., 2023, cited on the manuscript). We understand that it would be interesting to present it here, but due to the limitation on page numbers in the format “brief communication” it is not possible to present these results here.
10) We appreciate the comment for displaying the survey in supplementary, and we already provide them here in the attached material.
11) Concerning the comment of RC2 on the importance of piloting phase, the authors would also like to clarify that a pilot phase was made, testing the survey with a small group of experts, as discussed in lines 119 - 121. We are sorry that this wasn’t clear and we hope to be able to clarify this in the revised version of the manuscript.
12) We appreciate the comments made by RC2, and we hope we have addressed them all here in the reply and we hope to have the opportunity to make the corrections on the manuscript, to improve the final text of the brief communication.
-
AC2: 'Reply on RC2', Marina Batalini de Macedo, 07 Dec 2023
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2023-2042', Anonymous Referee #1, 20 Sep 2023
The paper offers a fascinating insight into the challenges of creating a global survey, even when this is aimed at experts. It is great to see researchers sharing their experiences, tips, and tricks; it is beneficial for the community when facing similar challenges. Nevertheless, the paper suffers from major limitations, both conceptually and structurally.
The paper presents itself as conceptually confused. First of all, the central topic of the survey, resilience, is poorly discussed. This is particularly crucial since the term is famously contentious, and slippery, and has been fought over by several disciplines. Instead, the researchers choose to define it using the Sendai framework's definition, which is itself vague and meant for high-level decision-makers. This definition comes from the UNISDR terminology dictionary (which is the collection of working definitions used by UNDRR), dating back to 2009. Since then, resilience has largely evolved as a term and has become the focus of much scholarly debate, itself unaddressed in the paper. The authors claim a rigid application of the framework, yet this is seemingly in contrast with the chosen framework and with what they present. This brings me to my next point, the lack of clarity on what resilience is is reflected in the indicators chosen for the index. By their own admission, the authors followed a vulnerability, hazard, and exposure framework, which is usually used to define risk. Hence, the author's definition of resilience is seemingly the inverse of risk. If this is the case, it should be discussed, as it can be the cause of the misunderstandings and resistance that the authors encountered when initially sharing their survey with the respondents. The authors fail to acknowledge how epistemological differences across disciplines can be at the root of the issue of defining resilience, instead indicating “significant variation between regions” (line 147) as the issue. Instead, the usefulness of resilience indicators and how they are seemingly hard to make is repeated several times without providing references.
Structurally the paper is uneven. First of all, the introduction gives an unclear idea of what the scope of the paper is. If your intention is to talk about the challenges and limitations of global surveys you should do that from the start. As it stands, the introduction talks about the importance of global resilience indexes. I suggest focussing from the get-go on the importance of global surveys, their challenges, and what the paper offers. Why resilience indexes are important is secondary. The fact that the authors created a resilience index should be addressed as long as it is used to show the challenges of creating a global index (the challenges of working with a slippery concept, for example), while the importance of a global resilience index is secondary. This can be addressed in two sentences with a good amount of references to support it. I believe that changing the focus of the introduction can also help improve the quality of the paper conceptually, by shifting the focus away from resilience and towards the survey itself. Write the whole introduction about global surveys and global indexes.
Finally, the authors make little use of references, especially in critical statements, such as the use of resilience indexes, and the presence of such indexes in the literature. Additionally, the terminology is often confusing and unclear. While the aim of the study is that of creating a resilience index, the authors refer to it as risk (lines 112, 146, 166), risk and resilience (lines 113, 136, 155). Are the two terms related in your analysis? If yes, state it clearly. Additionally, the paper opens with the statement “Drought is an omnipresent natural disaster”. The term natural disaster is generally considered very controversial (despite being used by the IPCC, for example) and might alienate readers with a background in DRR; I suggest disaster or natural hazard.
Overall, the paper contains very useful information on how to run a global survey of experts that I am grateful the authors decided to share with the community. I particularly appreciated the author’s commitment to highlighting the often-unaddressed a priori assumptions that go into making a questionnaire. Additionally, I found extremely useful the tips and tricks they shared, especially when specific (such as not having more than 40 questions, or keeping it within 15 minutes). I believe that by addressing the structural unevenness of the paper and reducing the focus on the resilience aspect itself, the authors will be able to create a piece of research worth sharing with the community. Thank you very much for your contributions.
Thank you!
-
AC1: 'Reply on RC1', Marina Batalini de Macedo, 07 Dec 2023
1) The authors would like to thank the comments from RC1, as they contribute to improving the quality of the text and how to present the main questions and contributions of this brief communication. Our motivation in writing this brief communication was the lack of papers or materials discussing the challenges of building global surveys, especially in tricky fields where we have conceptual divergences - such as resilience.
It is important to highlight that our research project has the main purpose of creating a decision support system (DSS) to help decision-makers evaluate the impact of different actions on decreasing vulnerability and improving the resilience of local farm communities to agricultural droughts. Creating a resilience composite index, with global representativeness, is one tool for our DSS. In this context, it was necessary to have input and perspectives from different experts across the globe, with different backgrounds, therefore we chose to do a global survey on drought indicators.
Even though this wasn’t the main part and contribution of our research project, this phase was the most time consuming and critical of our research. We spent approximately one year constructing the survey, due to the challenges presented in this brief communication. We believe that it is important to discuss the process of building the survey to prevent other researchers from passing the same problems that we had and improving the use and interpretation of this important tool. The importance of preparing the survey itself was also discussed by Elangovan and Sundaravel (2021) for any generic field. Here, we would like to complement the challenges in the resilience field.
References:
Elangovan and Sundaravel (2021). Method of preparing a document for survey instrument validation by experts. Available at: https://www.sciencedirect.com/science/article/pii/S2215016121001199
2) Regarding the comment related to the conceptual confusion regarding resilience, the authors agree with the RC1 and would like to have the opportunity to improve the text discussion on that. Resilience is indeed a contentious and slippery term, with different definitions for its quantification and components.This was actually our first main challenge, as described in lines 84 - 95. To create a global survey, replicable and interpretable, it was necessary to have a good definition of the resilience concept that we were using. The different concepts make it difficult to categorize the indicators in the resilience components and may affect the perception of their relevance to the respondent.
Therefore, we have decided to adopt the resilience term from a disaster risk reduction approach, as presented in the Sendai framework. We understand that this approach is meant for high-level decision-makers, but since the final aim of our research project is to create a decision support system for different international contexts, this definition was the most suitable for us. From the disaster risk reduction approach, it is important first to understand the evolution of how to deal with disasters, from an international and decision-making perspective.
Past discussions attempting to reduce the impacts of disasters on the community had a focus on disaster management, defined by the UNDRR (2021) as "the organization, planning and application of measures preparing for, responding to and recovering from disaster”. Therefore, the disaster management does not necessarily aim at averting or eliminating the threats, but decreasing the negative impacts resulting from the event and recovering as fast as possible to the original (or better) state of the system. Recently, new international discussions and documents on this topic have shifted from the approach of disaster management to disaster risk management. Disaster risk management is defined as “the application of disaster risk reduction policies and strategies to prevent new disaster risk, reduce existing disaster risk and manage residual risk, contributing to the strengthening of resilience and reduction of disaster losses” (UNDRR, 2021) aiming at actions in different timescales and with a final focus on increasing the economic, social, health and environmental resilience.
This change of approach in dealing with and managing disasters has been incorporated in the Sendai Framework for Disaster Risk Reduction 2015-2030 report (UNDRR, 2015). The report presents the importance of pre-disaster actions, such as prevention, mitigation, development and implementation of appropriate actions focused on preparing and effectively responding to disasters. To this end, it is emphasized the importance of risk assessment actions and dissemination of location-based disaster risk information, for a science-policy interface in the decision-making process in disaster risk management (called risk-informed-decision-making).
As previously mentioned, the ultimate goal of disaster risk management is to increase and strengthen resilience in its different aspects, with a greater focus on people's health and livelihood. The UNDDR (2021) defines resilience as “the ability of a system or community to anticipate, resist, prepare, respond to and recover from an event with multiple risks, with the least possible harm to social, economic, and environmental well-being”. For disaster risk reduction actions, it is important to establish concrete targets and to quantify the resilience from indicators and in different timeframes (UNDRR, 2015) for its components of speed, robustness, resourcefulness, and redundancy (Karamouz et al., 2016). Several indices have been proposed over the years to represent the level of resilience of a given system to a disruptive event. In general, the resilience assessment requires the identification of the risks present in the system due to the disruptive event and the risk management policy adopted by local agents to prevent its occurrence or reduce its impacts along the system's chain.
The risk analysis can be represented by a function that correlates the probability of occurrence of the disruptive event (H), the vulnerability of the different components of the studied system (V) and the exposure to risk of the different components (E), so that vulnerability and exposure quantitatively represent the potential impacts of the event on the system (Merz et al., 2014). Within the disaster risk management and risk-oriented-decision-making approach, as adopted in the Sendai report, the risk analysis stage is of fundamental importance and a precursor to the decision-making process.
To quantify the actions proposed during the risk management stage, it is important to evaluate the type of the proposed action (structural/measures and non-structural/instruments – Schanze, 2006), its temporal component (anticipation, prevention, mitigation, preparedness, response, recovery – UNDRR, 2015; Toklu, 2017), the magnitude of the impacts if the proposed action fails (high, medium and low). According to these components, the actions can be correlated with the different system capacities that help reduce the disaster risk and further impacts, therefore, improving resilience, such as adaptive capacity, coping capacity, and transformative capacity.
From our literature review, we have identified that indicators related to quantifying the hazard component of the agricultural drought risk were already well established. Therefore, our focus on this survey was to advance on identifying indicators related to the risk impacts (vulnerability and exposure) and the system's capacities to deal with the risk and increase resilience (adaptive, coping and transformative capacity).
We agree with the RC1 that all of this discussion on the conceptual framework and terminology should be incorporated into the manuscript since in its current form it is confusing. The discussion about the resilience framework was one of the most important parts of our research, which ended by choosing to adopt the resilience concepts from a disaster risk reduction approach, due to its use of the Sendai framework. We failed to synthesize this discussion in a clear format in the text, and we hope we can have the opportunity to make these adjustments, such as changing the term natural disaster only to disaster.
Finally, the potential ambiguity that resilience can bring to the respondents, the method adopted is as heuristically addressed as coevolutionary-conceived for unprecedented climate-driven extremes. Hence, our method can be adapted for the intercomparison of different case studies (i.e. Kreibich et al, 2022) and for sharing novel metrics of open-access datasets under F.A.I.R. principles (e.g. Kreibich et al, 2023).
References:
Karamouz, M., Rasoulnia, E., Zahmatkesh, Z., Olyaei, M. A., & Baghvand, A. (2016). Uncertainty-based flood resiliency evaluation of wastewater treatment plants. Journal of Hydroinformatics, 18(6), 990-1006.
Kreibich, H., et al. (2022) The challenge of unprecedented floods and droughts in risk management. Nature 608, 80–86 https://doi.org/10.1038/s41586-022-04917-5
Kreibich, H., et al (2023) Panta Rhei benchmark dataset: socio-hydrological data of paired events of floods and droughts, Earth System Science Data 15(5) https://doi.org/10.5194/essd-15-2009-2023
Merz, B., Hall, J., Disse, M., & Schumann, A. (2010). Fluvial flood risk management in a changing world. Natural Hazards and Earth System Sciences, 10(3), 509-527.
Toklu, A. (2017). Improving Organisational Performance with Balanced Scorecard in Humanitarian Logistics: A Proposal for Key Performance Indicators. International Journal of Academic Research in Accounting, Finance and Management Sciences, 7(1), 131-137.
UNDRR – United Nations Office for Disaster Risk Reduction. (2015). Sendai Framework for Disaster Risk Reduction 2015-2030.
UNDRR – United Nations Office for Disaster Risk Reduction. (2021). Terminology. Available in: < https://www.undrr.org/terminology>. Access on: Jun 2021.
3) As for the introduction part, we agree with RC1 that the focus should change from the importance of the global resilience index to a more focus on the construction of the survey itself. As discussed before, our research project's purpose is on using a global drought index as a tool for DSS and this led us to emphasize the importance of the index in the introduction. However, we agree with the RC1 that this index is not the focus of this brief communication. Therefore, in a revised version of the manuscript we will restructure the introduction to a more on point on the survey itself. Additionally, we agree with the reviewer that the term natural disaster may give a misunderstanding of the socio-natural-antropic mix that composes the disasters. Therefore, in the revised version we will change the term natural disaster only for disaster.4) Concerning the little use of references, we are sorry that we didn´t make more use of references in critical statements, but we should pay attention to the limitation of a maximum 20 references on brief communications. We believe it is important to prioritize the references used for each indicator selected on Table 1.
Despite that, we would like to provide here examples of the literature on resilience indices and global index that were part of our literature review to construct the survey and to support the use of resilience indexes, which can be included in the revised version of the brief communication.
One of the main literature on indices and indicators is provided by the Integrated Drought Management Programme (IDMP) (WMO and GWP, 2016), presenting the definition of drought impact as “an observable loss or change at a specific time because of drought”. In the handbook provided by them, they discuss the importance of using index/indices to help monitor, early warning and support decision-making for a risk-based drought management policy. Since drought is a many-sided disaster, using composite or hybrid indicators may be more representative. Additionally, this handbook provides guidelines for selecting indicators.
WMO and GWP (2016). Handbook of Drought Indicators and Indices. Available at: https://www.droughtmanagement.info/literature/GWP_Handbook_of_Drought_Indicators_and_Indices_2016.pdf
Other literature used in this study are provided bellow:
Alonso, Catarina, Celia M Gouveia, Ana Russo, and Patrícia Páscoa. 2019. "Crops' exposure, sensitivity and adaptive capacity to drought occurrence." Natural Hazards and Earth System Sciences 19 (12):2727-43.
Antwi-Agyei, Philip, Evan D. G. Fraser, Andrew J. Dougill, Lindsay C. Stringer, and Elisabeth Simelton. 2012. "Mapping the vulnerability of crop production to drought in Ghana using rainfall, yield and socioeconomic data." Applied Geography 32 (2):324-34.
Bachmair, Sophie, Kerstin Stahl, Kevin Collins, Jamie Hannaford, Mike Acreman, Mark Svoboda, Cody Knutson, Kelly Helm Smith, Nicole Wall, and Brian Fuchs. 2016. "Drought indicators revisited: the need for a wider consideration of environment and society." Wiley Interdisciplinary Reviews: Water 3 (4):516-36.
Blauhut, Veit. 2020. "The triple complexity of drought risk analysis and its visualisation via mapping: a review across scales and sectors." Earth-Science Reviews 210:103345.
Dabanli, Ismail. 2018. "Drought hazard, vulnerability, and risk assessment in Turkey." Arabian Journal of Geosciences 11 (18):1-12.
Dardonville, Manon, Christian Bockstaller, and Olivier Therond. 2021. "Review of quantitative evaluations of the resilience, vulnerability, robustness and adaptive capacity of temperate agricultural systems." Journal of Cleaner Production 286:125456.
Elagib, Nadir Ahmed. 2014. "Development and application of a drought risk index for food crop yield in Eastern Sahel." Ecological Indicators 43:114-25.
Fang, Dan, Zhuowei Hu, Zhiheng Wang, and Guangyao Duan. 2011. Drought disaster risk assessment and mapping at different scales based on RS and GIS. Paper presented at the 2011 19th International Conference on Geoinformatics.
Jayanthi, Harikishan, and Greg Husak. 2013. "A probabilistic approach to assess agricultural drought risk." Background Paper Prepared for the Global Assessment Report on Disaster Risk Reduction.
Murthy, C. S., Manoj Yadav, J. Mohammed Ahamed, B. Laxman, R. Prawasi, M. V. R. Sesha Sai, and R. S. Hooda. 2015. "A study on agricultural drought vulnerability at disaggregated level in a highly irrigated and intensely cropped state of India." Environmental Monitoring and Assessment 187 (3):140.
Xu, Huafeng, Kexin Xu, and Yingjie Yang. 2021. "Risk assessment model of agricultural drought disaster based on grey matter-element analysis theory." Natural hazards 107 (3):2693-707.
4) We appreciate the comments made by RC1 and we hope we have addressed them all here in the reply and we hope to have the opportunity to make the corrections to the manuscript, to improve the final text of the brief communication.
Citation: https://doi.org/10.5194/egusphere-2023-2042-AC1
-
AC1: 'Reply on RC1', Marina Batalini de Macedo, 07 Dec 2023
-
RC2: 'Comment on egusphere-2023-2042', Anonymous Referee #2, 09 Oct 2023
Dear authors
I respect the publication of challenging research and unsuccessful experiments. I agree that it is important to reflect on and write about so other researchers can overcome hurdles and avoid the same problems.However, the manuscript as is shows a lack of theoretical grounding in the survey setup and also contains quite some spelling mistakes and sloppiness. The latter can be solved easily, but I am not sure the former can be improved easily as the manuscript reports about a past stakeholder engagement.
About the presentation: the title contains two language errors: lessons learned and experiences. In the abstract, the plural of index should be indices and raising should be reaching, I presume.
Line 30+ is quite vague (I do not understand what is meant there) and in general this alinea misses quite some references in my opinion.
The introduction does not base the presented research well in existing literature. Significant studies on drought vulnerability including stakeholder engagement like Meza et al. Ahmadalipour et al. should have been explained and reflected on, showing how the presented study builds on it. (see also: European Commission, Joint Research Centre, Hagenlocher, M., Vogt, J., Meza, I. et al., Drought vulnerability indicators for global-scale drought risk assessments – Global expert survey results report, Publications Office, 2019, https://data.europa.eu/doi/10.2760/73844)
Line 41 misses a word "to"
I like the short description of multiple strategies to collect expert data but I would have loved to see references to authors who used these approaches in contexts relatable to the one described in the manuscript. Especially the followed Delphi technique should have been more elaborated and with references to relevant works.
Content wise, a better embedding in existing studies would have helped designing the study. For example, the use of the latest UNDRR terminology or the IPCC glossary on terms such as vulnerability sensitivity coping capacity and resilience would have helped starting discussions and creating a common understanding with the stakeholders. Now, the three components of risk (as also acknowledged in the Sendai Framework), hazard, exposure, vulnerability, are confused and mixed up both in the manuscript text and I fear also in the study. (e.g line 112)
I like the narrow focus for this study as in "the risk for whom / what" being small farms - I would have loved to see the full list of 136 indicators in supplementary (it is also unclear how this list was extracted and why it was not based on a structured literature review).
It remains confusing to me as to why indicators related to hazard are part of the resilience framework.
I appreciate that the authors realised that co-dependent variables would hamper the study and that too much indicators (or indices?) would not allow for a good survey. However, the next steps are not very transparent: How is the pre-selected list created (how many indicators are left?)? How are ratings discussed in the group discussions (and who was present there?) Why are relevancy, ease of understanding, accessibility, objectivity and consistency chosen as evaluation metrices (subjective choice? literature behind it?)?
Also here, references to show on which survey design choices are based, are lacking.
I am inspired by your approach to use publications in the web of science and scopus to reach scientists, but would have hoped that the authors had used existing policy networks around drought (IDMP, UNCCD, WMO, FAO) to reach other types of stakeholders (the do have databases of practitioners....).
I know getting responses is a real challenges and 120 is not too bad, but indeed the distribution in terms of sector, background and region is quite relevant. However, I would have loved to see some of the results / distribution analysis of the 120 respondents and also about the discussion group to understand exactly who was reached.
It would be good to disclose the survey in supplementary, as now it remains unclear whether for example open questions were asked to check for missing indices.
In general, there were a few critical steps from standard "how to built stakeholder surveys - literature" that were not followed (For example, I do not read anything about a piloting phase) hence the - to the authors - disappointing or challenging result.
Therefore, I feel this is a very interesting learning experience that mainly points to the relevance of using other studies, guidance frameworks, textbooks on qualitative research methods, when building a global survey.
However, I am not sure it is suitable in its current form for publication in NHESS.Citation: https://doi.org/10.5194/egusphere-2023-2042-RC2 -
AC2: 'Reply on RC2', Marina Batalini de Macedo, 07 Dec 2023
1) The authors would like to thank the comments from RC2, as it contributes to improving the quality of the text and how to present the main questions and contributions of this brief communication. Our motivation in writing this brief communication was the lack of papers or materials discussing the challenges of building global surveys, especially in tricky fields where we have conceptual divergences - such as resilience.
It is important to highlight that our research project has the main purpose of creating a decision support system (DSS) to help decision-makers evaluate the impact of different actions on decreasing vulnerability and improving the resilience of local farm communities to agricultural droughts. Creating a resilience composite index, with global representativeness, is one tool for our DSS. In this context, it was necessary to have input and perspective from different experts across the globe, with different backgrounds, therefore we chose to do a global survey on drought indicators.
Even though this wasn’t the main part and contribution of our research project, this phase was the most time consuming and critical of our research. We spent approximately one year constructing the survey, due to the challenges presented in this brief communication.
We believe that addressing the difficulties of the survey construction process is not the focus of the discussion of the papers for fear of the researchers, by sharing the problems obtained, weakening confidence in their final results presented later. However, we need to face this fear, as sharing challenges is also an important gain of knowledge. This was an issue found by other authors as well, e.g. Baker et al. (2014) states that “..While there is a rich literature on expert elicitation approaches and protocols, there is less information available on the specifics of how an elicitation is carried out".
Reference:
Baker et al. (2014). Facing the Experts: Survey Mode and Expert Elicitation. Available at: https://www.jstor.org/stable/pdf/resrep01013.pdf
2) The authors appreciate the RC2 for paying our attention to the English language errors. We appreciate your understanding that English is not the first language of the majority of the co-authors and sometimes this can lead to some errors, that we are glad it was pointed out before the final version of the brief communication. We will take a meticulous language review of the text.
3) The reviewer commented about important references that were not considered in our manuscript, like Meza et al. Ahmadalipour et al.. We would like to highlight that Meza et al. was used as a reference and we would like to have the opportunity to also incorporate the one by Ahmadalipour et al.
We would like to highlight that we are sorry that we didn´t make more use of references in critical statements, but we should pay attention to the limitation of a maximum 20 references on brief communications. We believe it is important to prioritize the references used for each indicator selected on Table 1.
Despite that, we would like to provide here examples of the literature on resilience indices and global index that were part of our literature review to construct the survey and to support the use of resilience indexes, which can be included in the revised version of the brief communication.
One of the main literature on indices and indicators is provided by the Integrated Drought Management Programme (IDMP) (WMO and GWP, 2016), presenting the definition of drought impact as “an observable loss or change at a specific time because of drought”. In the handbook provided by them, they discuss the importance of using index/indices to help monitor, early warning and support decision-making for a risk-based drought management policy. Since drought is a many-sided disaster, using composite or hybrid indicators may be more representative. Additionally, this handbook provides guidelines for selecting indicators.
WMO and GWP (2016). Handbook of Drought Indicators and Indices. Available at: https://www.droughtmanagement.info/literature/GWP_Handbook_of_Drought_Indicators_and_Indices_2016.pdf
Other literature used in this study are provided bellow:
Alonso, Catarina, Celia M Gouveia, Ana Russo, and Patrícia Páscoa. 2019. "Crops' exposure, sensitivity and adaptive capacity to drought occurrence." Natural Hazards and Earth System Sciences 19 (12):2727-43.
Antwi-Agyei, Philip, Evan D. G. Fraser, Andrew J. Dougill, Lindsay C. Stringer, and Elisabeth Simelton. 2012. "Mapping the vulnerability of crop production to drought in Ghana using rainfall, yield and socioeconomic data." Applied Geography 32 (2):324-34.
Bachmair, Sophie, Kerstin Stahl, Kevin Collins, Jamie Hannaford, Mike Acreman, Mark Svoboda, Cody Knutson, Kelly Helm Smith, Nicole Wall, and Brian Fuchs. 2016. "Drought indicators revisited: the need for a wider consideration of environment and society." Wiley Interdisciplinary Reviews: Water 3 (4):516-36.
Blauhut, Veit. 2020. "The triple complexity of drought risk analysis and its visualisation via mapping: a review across scales and sectors." Earth-Science Reviews 210:103345.
Dabanli, Ismail. 2018. "Drought hazard, vulnerability, and risk assessment in Turkey." Arabian Journal of Geosciences 11 (18):1-12.
Dardonville, Manon, Christian Bockstaller, and Olivier Therond. 2021. "Review of quantitative evaluations of the resilience, vulnerability, robustness and adaptive capacity of temperate agricultural systems." Journal of Cleaner Production 286:125456.
Elagib, Nadir Ahmed. 2014. "Development and application of a drought risk index for food crop yield in Eastern Sahel." Ecological Indicators 43:114-25.
Fang, Dan, Zhuowei Hu, Zhiheng Wang, and Guangyao Duan. 2011. Drought disaster risk assessment and mapping at different scales based on RS and GIS. Paper presented at the 2011 19th International Conference on Geoinformatics.
Jayanthi, Harikishan, and Greg Husak. 2013. "A probabilistic approach to assess agricultural drought risk." Background Paper Prepared for the Global Assessment Report on Disaster Risk Reduction.
Murthy, C. S., Manoj Yadav, J. Mohammed Ahamed, B. Laxman, R. Prawasi, M. V. R. Sesha Sai, and R. S. Hooda. 2015. "A study on agricultural drought vulnerability at disaggregated level in a highly irrigated and intensely cropped state of India." Environmental Monitoring and Assessment 187 (3):140.
Xu, Huafeng, Kexin Xu, and Yingjie Yang. 2021. "Risk assessment model of agricultural drought disaster based on grey matter-element analysis theory." Natural hazards 107 (3):2693-707.
4) Concerning the suggestion of RC2 on the Delphi technique and more elaboration with references to relevant work, we would like to justify this because the manuscript type chosen by the group to present the challenges faced by us on constructing the global survey is the brief communications. Unfortunately, this type of manuscript does not allow more than 20 references (https://www.natural-hazards-and-earth-system-sciences.net/about/manuscript_types.html), which limited our possibilities in including on the manuscript authors that used these approaches in the drought resilience context. We have prioritized the references related to the indicators/indices selected by the authors to be part of the survey (presented in Table 1).
Despite this limitation, we would like to present here some examples of papers that used these different approaches:
Proposal for a Tool for Assessing Access to Water in Rural Communities: a Case Study in the Brazilian Semi-arid. https://link.springer.com/article/10.1007/s00267-022-01600-3
Select indicators and prioritize solutions for desertification and drought in Binh Thuan, Vietnam. https://www.tandfonline.com/doi/abs/10.1080/10042857.2016.1177315
5) The resilience framework chosen for us on this research project was based on the disaster risk reduction approach, as presented in the Sendai framework. From the disaster risk reduction approach, it is important first to understand the evolution of how to deal with disasters, from an international and decision-making perspective.
Disaster risk management is defined as “the application of disaster risk reduction policies and strategies to prevent new disaster risk, reduce existing disaster risk and manage residual risk, contributing to the strengthening of resilience and reduction of disaster losses” (UNDRR, 2021) aiming at actions in different timescales and with a final focus on increasing the economic, social, health and environmental resilience.
This approach in dealing with and managing disasters has been incorporated in the Sendai Framework for Disaster Risk Reduction 2015-2030 report (UNDRR, 2015). The report presents the importance of pre-disaster actions, such as prevention, mitigation, development and implementation of appropriate actions focused on preparing and effectively responding to disasters. To this end, it is emphasized the importance of risk assessment actions and dissemination of location-based disaster risk information, for a science-policy interface in the decision-making process in disaster risk management (called risk-informed-decision-making).
As previously mentioned, the ultimate goal of disaster risk management is to increase and strengthen resilience in its different aspects, with a greater focus on people's health and livelihood. The UNDDR (2021) defines resilience as “the ability of a system or community to anticipate, resist, prepare, respond to and recover from an event with multiple risks, with the least possible harm to social, economic, and environmental well-being”. For disaster risk reduction actions, it is important to establish concrete targets and to quantify the resilience from indicators and in different timeframes (UNDRR, 2015) for its components of speed, robustness, resourcefulness, and redundancy (Karamouz et al., 2016). Several indices have been proposed over the years to represent the level of resilience of a given system to a disruptive event. In general, the resilience assessment requires the identification of the risks present in the system due to the disruptive event and the risk management policy adopted by local agents to prevent its occurrence or reduce its impacts along the system's chain.
The risk analysis can be represented by a function that correlates the probability of occurrence of the disruptive event (H), the vulnerability of the different components of the studied system (V) and the exposure to risk of the different components (E), so that vulnerability and exposure quantitatively represent the potential impacts of the event on the system (Merz et al., 2014). Within the disaster risk management and risk-oriented-decision-making approach, as adopted in the Sendai report, the risk analysis stage is of fundamental importance and a precursor to the decision-making process.
From our literature review, we have identified that indicators related to quantifying the hazard component of the agricultural drought risk were already well established. Therefore, our focus on this survey was to advance on identifying indicators related to the risk impacts (vulnerability and exposure) and the system's capacities to deal with the risk and increase resilience (adaptive, coping and transformative capacity).
References:
Karamouz, M., Rasoulnia, E., Zahmatkesh, Z., Olyaei, M. A., & Baghvand, A. (2016). Uncertainty-based flood resiliency evaluation of wastewater treatment plants. Journal of Hydroinformatics, 18(6), 990-1006.
Merz, B., Hall, J., Disse, M., & Schumann, A. (2010). Fluvial flood risk management in a changing world. Natural Hazards and Earth System Sciences, 10(3), 509-527.
UNDRR – United Nations Office for Disaster Risk Reduction. (2015). Sendai Framework for Disaster Risk Reduction 2015-2030.
UNDRR – United Nations Office for Disaster Risk Reduction. (2021). Terminology. Available in: < https://www.undrr.org/terminology>. Access on: Jun 2021.
6) Regarding the list of the 136 indicators, the initial selection of these indicators was done from a structured literature review, as mentioned in lines 103 and 104. I am sorry if this information wasn’t clear in the text. In the revised version we can make it clearer. We appreciate the comment for displaying the full list of 136 indicators in supplementary, and we already provide them here in the attached material.
Further, the selection of the final list of indicators was made through three steps, related to the issues presented in lines 103 - 115. The first step was removing hazard indicators. In this step, 31 indicators were removed. In the second step, we removed co-dependent indicators/indices from the list, remaining the ones with more availability and easy to get. E.g. from Gini index and poverty rate, we remain with the poverty rate, since it is a more direct measurement and easier to get in different contexts. This process of eliminating co-dependent indicators/indices was made interactively in group discussion sessions. 28 indicators were removed in this process. The third step was reducing the total number of indicators/indices to avoid the survey becoming too extensive and exhaustive to answer. From this part, each participant of the group (in total 10 participants, distributed from Brazil, UK, and USA, but with different backgrounds on their birth countries and graduation courses) independently has evaluated the relevance of the indicators/indices, through a form available only for the group. From the answers in that form, in a group discussion session we have selected the 33 indicators with the higher average rating. 3 indicators were later added after the first pilot run, as they were suggested by experts on this pilot.
Additionally, during the discussion sessions with the relevance data, one of the concerns was that some indicators/indices are very interesting and relevant, but they are not easy to obtain. Therefore, it was also interesting to consider other qualitative aspects when selecting the final indicators and weights for the composite drought index, such as ease of understanding, accessibility, and objectivity. The choice of these specific metrics came from the literature (Sweya et al., 2021). The paper presented 5 complementary attributes for social resilience indicators of water supply systems (which are: affordability, availability, reliability, simplicity, and transparency). As a result, they have obtained that data availability, reliability and affordability were the most limiting factors for selecting the indicators in Tanzania. In this sense, and with a focus on the Global South, the group selected the three me metrics before mentioned to be complementary to the relevancy and adapted from Sweya et al. (2021), where understanding was used to represent the simplicity, accessibility was used as a single attribute to account for affordability and availability and objectivity was an additional attribute that we chosen to evaluate how objective is the final measure (since come of our social indicators are political measurements).
Reference:
Sweya et al. (2021). A social resilience measure tool for Tanzania’s water supply systems. Available at: https://www.sciencedirect.com/science/article/pii/S2212420921005197
7) Concerning the survey design, this was made based on guidelines for operationalizing the Delphi method (Hasson et al., 2000) and the suggestions made by Elangovan and Sundaravel (2021) on preparing documents for survey instrument validation by experts. These authors state that: “Even though there are many literatures available on the reliability and validity procedures, many researches struggle to operationalize the process.”, therefore providing a template to validate the survey instrument. However, they present a generic form of document, in which we still experienced difficulties related to the resilience field study.
Therefore, additionally, we have improved our survey design based on the evaluation of different literature that used Delphi method to access resilience indicators (e.g. Alshehri et al., 2015; Ogah et al., 2021), identifying the main problems obtained, literature on composite index for resilience (Sweya et al.; 2021), in which we could identify important complementary questions about attributes that usually are not asked in the surveys and all the relevant data are assumed to be equally easy to obtain and understand.
During this one year of constructing the survey, different survey design were tested within the group and they were being adapted based on the user experience.
We are sorry that we didn’t explain this in the text and we hope it is clear now. We would like to have the opportunity to incorporate a discussion specifically on the design on the revised version of the manuscript.
References:
Hasson et al. (2000). Research guidelines for the Delphi survey technique. Available at: https://pubmed.ncbi.nlm.nih.gov/11095242/
Elangovan and Sundaravel (2021). Method of preparing a document for survey instrument validation by experts. Available at: https://www.sciencedirect.com/science/article/pii/S2215016121001199
Alsherhri et al. (2015). Delphi-based consensus study into a framework of community resilience to disaster. Available at: https://link.springer.com/article/10.1007/s11069-014-1423-x
Ogah et al. (2021). Operationalising Community Resilience to Climate Change in Developing Countries: A Grounded Delphi Method (GDM) Approach. Available at: https://assets.researchsquare.com/files/rs-844800/v1/a597c71b-b9e8-4f50-a1b5-f7bcc3a645fd.pdf?c=1643475669
Sweya et al. (2021). A social resilience measure tool for Tanzania’s water supply systems. Available at: https://www.sciencedirect.com/science/article/pii/S2212420921005197
8) As for the suggestion of the RC2 on searching other types of stakeholders within the databases of practitioners, we appreciate this suggestion very much. We will try to contact these policy networks to have access to the database and maybe do a second round of the survey. We will also incorporate this suggestion in the revised version of the manuscript.
9) Regarding the comment of RC2 on presenting results on distribution analysis on the 120 respondents, the focus of this brief communication was to present the main challenges on constructing a global survey a priori. The results obtained by the survey, such as the distribution of the 120 respondents, are presented in an additional paper (Sass et al., 2023, cited on the manuscript). We understand that it would be interesting to present it here, but due to the limitation on page numbers in the format “brief communication” it is not possible to present these results here.
10) We appreciate the comment for displaying the survey in supplementary, and we already provide them here in the attached material.
11) Concerning the comment of RC2 on the importance of piloting phase, the authors would also like to clarify that a pilot phase was made, testing the survey with a small group of experts, as discussed in lines 119 - 121. We are sorry that this wasn’t clear and we hope to be able to clarify this in the revised version of the manuscript.
12) We appreciate the comments made by RC2, and we hope we have addressed them all here in the reply and we hope to have the opportunity to make the corrections on the manuscript, to improve the final text of the brief communication.
-
AC2: 'Reply on RC2', Marina Batalini de Macedo, 07 Dec 2023
Peer review completion
Journal article(s) based on this preprint
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 399 | 102 | 37 | 538 | 28 | 31 |
- HTML: 399
- PDF: 102
- XML: 37
- Total: 538
- BibTeX: 28
- EndNote: 31
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
Marina Batalini de Macedo
Marcos Roberto Benso
Karina Simone Sass
Adelaide Cassia Nardocci
Eduardo Mario Mendiondo
Greicelene Jesus da Silva
Pedro Gustavo Câmara da Silva
Abdullah Konak
Nazmiye Balta-Ozkan
Michael Jacobson
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(621 KB) - Metadata XML