Understanding representations of uncertainty, an eye-tracking study part II: The effect of expertise

Williams, Louis; Mulder, Kelsey; Charlton-Perez, Andrew; Lickiss, Matthew; Black, Alison; McCloy, Rachel; McSorley, Eugene; Young, Joe

doi:https://doi.org/10.5194/egusphere-2022-929

Preprints

https://doi.org/10.5194/egusphere-2022-929

Preprints

10 Nov 2022

| 10 Nov 2022

Understanding representations of uncertainty, an eye-tracking study part II: The effect of expertise

Louis Williams, Kelsey Mulder, Andrew Charlton-Perez, Matthew Lickiss, Alison Black, Rachel McCloy, Eugene McSorley, and Joe Young

Abstract. As the ability to make predictions of uncertainty information representing natural hazards increases, an important question for those designing and communicating hazard forecasts is how visualisations of uncertainty influence understanding amongst the intended, potentially varied, target audiences. End-users have a wide range of differing expertise and backgrounds, possibly influencing the decision-making process they undertake for a given forecast presentation. Our previous, linked study, examined how the presentation of uncertainty information influenced end-user decision making. Here, we shift the focus to examine the decisions and reactions of participants with differing expertise (Meteorology, Psychology and Graphic Communication students) when presented with varied hypothetical forecast representations (boxplot, fan plot or spaghetti plot with and without median lines), using the same eye-tracking methods and experiments. Participants made decisions about a fictional scenario involving the choices between ships of different sizes in the face of varying ice thickness forecasts. Eye-movements to the graph area and key, and how they changed over time (early, intermediate, and later viewing periods), were examined. More fixations (maintained gaze on one location) and time fixating was spent on the graph and key during early and intermediate periods of viewing, particularly for boxplots and fan plots. The inclusion of median lines led to less fixations being made to all graph types during early and intermediate viewing periods. No difference in eye movement behaviour was found due to expertise, however those with greater expertise were more accurate in their decisions, particularly during more difficult scenarios. Where scientific producers seek to draw users to the central estimate, an anchoring line can significantly reduce cognitive load leading both experts and non-experts to make more rational decisions. When asking users to consider extreme scenarios or uncertainty, different prior expertise can lead to significantly different cognitive load for processing information with an impact on ability to make appropriate decisions.

Received: 14 Sep 2022 – Discussion started: 10 Nov 2022

Download & links

Preprint (PDF, 429 KB)

Notice on discussion status
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
Preprint (429 KB)

Download & links

The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.

Journal article(s) based on this preprint

06 Sep 2023

Understanding representations of uncertainty, an eye-tracking study – Part 2: The effect of expertise

Louis Williams, Kelsey J. Mulder, Andrew Charlton-Perez, Matthew Lickiss, Alison Black, Rachel McCloy, Eugene McSorley, and Joe Young

Geosci. Commun., 6, 111–123, https://doi.org/10.5194/gc-6-111-2023,https://doi.org/10.5194/gc-6-111-2023, 2023

Short summary

Louis Williams et al.

Interactive discussion

Status: closed

RC1:
'Comment on egusphere-2022-929', Anonymous Referee #1, 05 Jan 2023

This paper is linked/an extension to a previous study, but it does have a novel aspect and there is a clear rationale for the study.

I feel there needs to be more dicussion examining the decisions linked to expertise. I feel the findings need to be slighlty toned down to reflect the lack of signficant differences in decision making capacity relative to expertise as there was a difference at 50% but not 30 or 70%.

It may be minor but introducing someone else's results (line 238, 279) in your results section is quite confusing

Also relative to the experts and non experts, I would agree with the comments made in line 336 considering the recruitment of true experts in the area as a comparison. It would also help to link back to why expertise had little impact on eye movement behaviour (line 341)

Citation: https://doi.org/10.5194/egusphere-2022-929-RC1
- AC1: 'Reply on RC1', Eugene McSorley, 18 Mar 2023
  
  The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2022/egusphere-2022-929/egusphere-2022-929-AC1-supplement.pdf
  
  Citation: https://doi.org/10.5194/egusphere-2022-929-AC1
RC2:
'Comment on egusphere-2022-929', Christopher Chagumaira, 21 Feb 2023
I enjoyed reading the manuscript, however there are some sections of the paper that need strenghtening. As a first comment, I would like to see this paper a stand alone and most of the times the authors make reference to another paper (which is not yet published). My specific comments are as follows:
L48 Not clear how Covid19 further impacted communication of uncertainty.

L56 The problem is not well articulated and the readers would benefit from an extensive literature search of how end-users have struggled with communication of uncertainty. A lot of work has been done in the past and the authors can make reference to how some the challenges to communcating uncertainty (eg. Spiegelhalter, D., Pearson, M., and Short, I.: Visualizing Uncertainty About the Future, Science, 333, 1393–1400, https://doi.org/10.1126/science.1191181, 2011).

L78-- L85 A lot has been done on this subject of using visuals for communicating uncertainty and this would be important to emphasize this in this section (see Milne, A. E., Glendining, M. J., Lark, R. M., Perryman, S. A. M., Gordon, T., and Whitmore, A. P.: Communicating the uncertainty in estimated greenhouse gas emissions from agriculture, J. Environ. Manage., 160, 139–153, https://doi.org/10.1016/j.jenvman.2015.05.034, 2015; Zikmund-Fisher, B. J., Fagerlin, A., and Ubel, P. A.: Improving Understanding of Adjuvant Therapy Options by Using Simpler Risk Graphics, Cancer, 113, 3382–3390, https://doi.org/10.1002/cncr.23959, 2008)

L112-- L122 I dont think this is very necessary to discuss the relation between the two papers at length, I would focus on the problem this manuscript is focusing on and why it is important. The authors introduce the concept of eye-movement but quickly divert to another paper. Focus should remain on this paper. Again I would expand on the complex methods the authors mention in passsing and why they are relevant. A supplement of the test methods would be useful.

L125 i dont see how these research questions analyse/capture the differences in the end-user groups

L147 Here again the authors mention the companion paper (which cant be found anywhere)- A summary of the methodolgy would be useful.

L153 a graphical illustration would be more useful here and an explanation why the Eye link II eye tracker is useful or important on L156 is needed.

I agree with the first referee, why would you report results from another paper, this paper should stand alone and casually make reference to the other paper when needed.

L193 to 198 is the justification and it seems misplaced.

The information presented from L199 to 204 should be presented in a table. Similar comment for L211 to 223.

L238 seems misplaced as well as it belongs to discussion

L259 to 266 should be presented in a table.

L290 A plot of the accuracies for the different clusters would communicate the research findings here!
Citation: https://doi.org/10.5194/egusphere-2022-929-RC2
- AC2: 'Reply on RC2', Eugene McSorley, 18 Mar 2023
  
  The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2022/egusphere-2022-929/egusphere-2022-929-AC2-supplement.pdf
  
  Citation: https://doi.org/10.5194/egusphere-2022-929-AC2

Interactive discussion

Status: closed

RC1:
'Comment on egusphere-2022-929', Anonymous Referee #1, 05 Jan 2023

This paper is linked/an extension to a previous study, but it does have a novel aspect and there is a clear rationale for the study.

I feel there needs to be more dicussion examining the decisions linked to expertise. I feel the findings need to be slighlty toned down to reflect the lack of signficant differences in decision making capacity relative to expertise as there was a difference at 50% but not 30 or 70%.

It may be minor but introducing someone else's results (line 238, 279) in your results section is quite confusing

Also relative to the experts and non experts, I would agree with the comments made in line 336 considering the recruitment of true experts in the area as a comparison. It would also help to link back to why expertise had little impact on eye movement behaviour (line 341)

Citation: https://doi.org/10.5194/egusphere-2022-929-RC1
- AC1: 'Reply on RC1', Eugene McSorley, 18 Mar 2023
  
  The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2022/egusphere-2022-929/egusphere-2022-929-AC1-supplement.pdf
  
  Citation: https://doi.org/10.5194/egusphere-2022-929-AC1
RC2:
'Comment on egusphere-2022-929', Christopher Chagumaira, 21 Feb 2023
I enjoyed reading the manuscript, however there are some sections of the paper that need strenghtening. As a first comment, I would like to see this paper a stand alone and most of the times the authors make reference to another paper (which is not yet published). My specific comments are as follows:
L48 Not clear how Covid19 further impacted communication of uncertainty.

L56 The problem is not well articulated and the readers would benefit from an extensive literature search of how end-users have struggled with communication of uncertainty. A lot of work has been done in the past and the authors can make reference to how some the challenges to communcating uncertainty (eg. Spiegelhalter, D., Pearson, M., and Short, I.: Visualizing Uncertainty About the Future, Science, 333, 1393–1400, https://doi.org/10.1126/science.1191181, 2011).

L78-- L85 A lot has been done on this subject of using visuals for communicating uncertainty and this would be important to emphasize this in this section (see Milne, A. E., Glendining, M. J., Lark, R. M., Perryman, S. A. M., Gordon, T., and Whitmore, A. P.: Communicating the uncertainty in estimated greenhouse gas emissions from agriculture, J. Environ. Manage., 160, 139–153, https://doi.org/10.1016/j.jenvman.2015.05.034, 2015; Zikmund-Fisher, B. J., Fagerlin, A., and Ubel, P. A.: Improving Understanding of Adjuvant Therapy Options by Using Simpler Risk Graphics, Cancer, 113, 3382–3390, https://doi.org/10.1002/cncr.23959, 2008)

L112-- L122 I dont think this is very necessary to discuss the relation between the two papers at length, I would focus on the problem this manuscript is focusing on and why it is important. The authors introduce the concept of eye-movement but quickly divert to another paper. Focus should remain on this paper. Again I would expand on the complex methods the authors mention in passsing and why they are relevant. A supplement of the test methods would be useful.

L125 i dont see how these research questions analyse/capture the differences in the end-user groups

L147 Here again the authors mention the companion paper (which cant be found anywhere)- A summary of the methodolgy would be useful.

L153 a graphical illustration would be more useful here and an explanation why the Eye link II eye tracker is useful or important on L156 is needed.

I agree with the first referee, why would you report results from another paper, this paper should stand alone and casually make reference to the other paper when needed.

L193 to 198 is the justification and it seems misplaced.

The information presented from L199 to 204 should be presented in a table. Similar comment for L211 to 223.

L238 seems misplaced as well as it belongs to discussion

L259 to 266 should be presented in a table.

L290 A plot of the accuracies for the different clusters would communicate the research findings here!
Citation: https://doi.org/10.5194/egusphere-2022-929-RC2
- AC2: 'Reply on RC2', Eugene McSorley, 18 Mar 2023
  
  The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2022/egusphere-2022-929/egusphere-2022-929-AC2-supplement.pdf
  
  Citation: https://doi.org/10.5194/egusphere-2022-929-AC2

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision | EF: Editorial file upload

ED: Reconsider after major revisions (further review by editor and referees) (23 Mar 2023) by Stephanie Zihms

AR by Eugene McSorley on behalf of the Authors (23 Mar 2023) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (30 Mar 2023) by Stephanie Zihms

RR by Anonymous Referee #2 (03 Apr 2023)

RR by Laura Carey (21 Apr 2023)

Suggestions for revision or reasons for rejection

The additional information added to the introduction does help to provide a clearer rationale and novel/standalone aspect to this paper. Although, the paper discusses scientific information with uncertainty in the introduction (highlighting COVID_19) and then also includes more specific information on geoscience and natural hazard data in the introduction/discussion. Does the context matter and if so how is the information on COVID_19 relevant and what does that mean for expertise and audience who are interpreting the data? Arguably the cohort in this sample have a degree of familiarity with interpreting data (albeit not in the context) as students in Higher Education and this may not cover the spectrum of ability represented in the introduction and discussion as the audience to the COVID_19 and non-specialists. In line 74 the authors discuss the “desired outcomes of the communicator and the needs and abilities of the audience” and I think this information needs to be considered in the context of the participants and the figures presented in this study.
There are also some points of clarification in the results and discussion
• Results
o For example, in line 344 and 345 in section 3.2 - the wording (In order to examine fixation to the key over different periods of the decision-making process") could be clarified and explained especially in relation to the sub heading of 3.2 title
o Also in 3.3 the results section starts by explaining the results of the companion paper.
• Discussion
o In Line 417- 418 ”More economically rational responses to the ship decision were made by meteorology students (greater level of expertise) during the most difficult scenarios”- a greater explanation for this finding is required, especially as eye movements did not change. In line 444 the authors refer to meteorology students "use their expertise". What expertise are they using and can this expertise be transferable or what else needs to communicated to non-specialised audience to help with interpretation if expertise is context specific?
Also the terms companion paper and the reference is used interchangeably throughout despite the text saying from X point it would be referred to as companion paper.

Hide

ED: Reconsider after major revisions (further review by editor and referees) (25 Apr 2023) by Stephanie Zihms

AR by Eugene McSorley on behalf of the Authors (26 Apr 2023) Author's response Author's tracked changes Manuscript

ED: Reconsider after major revisions (further review by editor and referees) (02 May 2023) by Stephanie Zihms

ED: Referee Nomination & Report Request started (01 Jun 2023) by Stephanie Zihms

RR by Anonymous Referee #2 (07 Jun 2023)

RR by Laura Carey (28 Jun 2023)

ED: Publish as is (04 Jul 2023) by Stephanie Zihms

ED: Publish subject to minor revisions (further review by editor) (18 Jul 2023) by Solmaz Mohadjer (Executive editor)

Dear Louis (and all co-authors),

Thank you very much for revising your manuscript based on the reviewers' comments. There are a few more edits I'd like to ask for before accepting your manuscript for publication. The main reason for asking these edits is to increase the readability of your work.

1. Your manuscript can benefit significantly from adding at least two figures (method section) and an additional table (result section). This was also pointed out by the reviewers. Please add a figure showing ice thickness forecasts in seven different types. This should help readers visualize the different graph types you use with your research participants. The second figure that would be useful is a photo (or cartoon drawing) of the eye tracking device together with a screenshot of what participants look at and how the eye tracking appears on the screen. Please note that including a link to the device website does not suffice as links can change or be removed over time. It also keeps the reader on the manuscript page instead of sending them to another page.

2. Sections 3.1 - 3.3 (results) are difficult to follow. Please list all the stats in a table that you can refer to in the text. This was also mentioned by the reviewers. Consider including an image of each graph in this table to help the readers identify quickly what data go with what graph type/area, and provide a caption that explains all the terminologies, and abbreviations you use in the table and text (e.g., F, MSE, etc.). Not all GC readers are familiar with these terms.

3. Please explain your stat work (mixed measure ANOVA - four-way vs. one-way) in the methods section and explain why this approach was chosen. Not every reader is familiar with this method. Also, consider moving line 294-296 to the methods section.

4. You have measured fixation frequency and duration. How about eye blinking and other types of eye movements that may be important to consider in the interpretation of your results. Are these measurable, and if so, will they be relevant to the conclusions of this study?

Thank you again. I look forward to seeing how you implement these changes.

Best,
Solmaz
GC Executive Editor

Hide

AR by Eugene McSorley on behalf of the Authors (19 Jul 2023) Author's response Author's tracked changes Manuscript

EF by Lorena Grabowski (20 Jul 2023)

ED: Publish subject to technical corrections (20 Jul 2023) by Solmaz Mohadjer

Thank you for your latest edits and adding the requested tables and figures. I'm happy to accept for publication subject to corrections shown below:

Line 230: Please consider deleting the first sentence in section 2.2 and reference the paper in the text: Full methodological details are given in our companion paper, but to restate the core procedure.

Lines 252-254: Please revise the sentence to "Participants wore an EyeLink II (SR Research Ltd) eye tracker headset (Fig 2) which recorded eye movements of the right eye at a rate of 500Hz as they completed the task" and delete the link to the product.

Line 257: Please explain what "0.5 deg average accuracy and 0.01 deg resolution" mean in the context of your study.

Line 262: Please insert "Fig 2" at the end of sentence "[...] by a chin rest."

Line 291: Please insert "(also known as ANOVA)" after Analysis of Variance.

Lines 301-316: Please move these lines to the caption for Table 1, and correct typos in lines 315-316.

Line 314: Typo (Eta-squared).

Line 328 onward: Not sure how useful it is to report all the stats in the text. It can be distracting. I wonder if referring to Table 1 suffices (also see comment below about Table 1).

Table 1: Thanks for putting this together. However, it is still not quite easy to read, understand, and compare data for different graph types. Adding more columns may fix this issue. For example, consider creating a separate column for each of these values: F, df, MSE, M, and Eta-squared.

Table 2: Same comment as above.

Line 473: Typo (demonstrate)

Line 475: Please delete 'discussed here'

Line 481: Typo (support) - Also lines 481-483 are complex and vague, please consider revision.

References: Please add a "period" after "al." for all references cited in your manuscript. Example: (Locher et al., 2007) and for those with two authors, replace '&' with 'and'.

Figure 1 and 2: Great additions. Thanks for putting these together. In Fig 2 caption, you state "Artefacts of this centering can be seen on the heat maps." Could you please add a label to the figure to indicate this? I am not sure if it can be seen by everyone. I also suggest deleting "Note that the small diagonal line visible on the top right of the display screen (bottom left photo) is an artefact of the photograph and the refresh rate of the monitor." It might be more confusing to mention it. It may be safe to assume that most people won't even recognize this. Just a suggestion.

Best,
Solmaz Mohadjer
(GC Executive Editor)

Hide

ED: Publish subject to minor revisions (further review by editor) (20 Jul 2023) by Solmaz Mohadjer (Executive editor)

AR by Eugene McSorley on behalf of the Authors (24 Jul 2023) Author's response Author's tracked changes Manuscript

EF by Polina Shvedko (25 Jul 2023)

ED: Publish as is (25 Jul 2023) by Solmaz Mohadjer

ED: Publish as is (25 Jul 2023) by Solmaz Mohadjer (Executive editor)

AR by Eugene McSorley on behalf of the Authors (26 Jul 2023) Manuscript

Journal article(s) based on this preprint

06 Sep 2023

Understanding representations of uncertainty, an eye-tracking study – Part 2: The effect of expertise

Louis Williams, Kelsey J. Mulder, Andrew Charlton-Perez, Matthew Lickiss, Alison Black, Rachel McCloy, Eugene McSorley, and Joe Young

Geosci. Commun., 6, 111–123, https://doi.org/10.5194/gc-6-111-2023,https://doi.org/10.5194/gc-6-111-2023, 2023

Short summary

Louis Williams et al.

Viewed

Total article views: 453 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	BibTeX	EndNote
324	113	16	453	4	4

HTML: 324
PDF: 113
XML: 16
Total: 453
BibTeX: 4
EndNote: 4

Views and downloads (calculated since 10 Nov 2022)

Month	HTML	PDF	XML	Total
Nov 2022	107	13	5	125
Dec 2022	73	8	0	81
Jan 2023	30	9	2	41
Feb 2023	35	14	2	51
Mar 2023	34	20	5	59
Apr 2023	13	6	0	19
May 2023	10	11	0	21
Jun 2023	6	12	1	19
Jul 2023	9	10	1	20
Aug 2023	6	10	0	16
Sep 2023	1	0	1
Oct 2023	0

Cumulative views and downloads (calculated since 10 Nov 2022)

Month	HTML	PDF	XML	Total
Nov 2022	107	13	5	125
Dec 2022	73	8	0	81
Jan 2023	30	9	2	41
Feb 2023	35	14	2	51
Mar 2023	34	20	5	59
Apr 2023	13	6	0	19
May 2023	10	11	0	21
Jun 2023	6	12	1	19
Jul 2023	9	10	1	20
Aug 2023	6	10	0	16
Sep 2023	1	0	1
Oct 2023	0

Viewed (geographical distribution)

Total article views: 440 (including HTML, PDF, and XML) Thereof 440 with geography defined and 0 with unknown origin.

Country	#	Views	%

Cited

Latest update: 07 Oct 2023

Download

The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.

Preprint (429 KB)
Metadata XML

Short summary

When constructing graphical environmental forecasts involving uncertainty it is important to consider the background and expertise of end-users. Using novel eye-tracking methods, we show that where people look and the decisions they make are both strongly influenced by prior expertise and the graphical construction of forecast representations common in presentations of environmental data. We suggest that forecasters should construct their presentations carefully bearing these factors in mind.


Total:	0
HTML:	0
PDF:	0
XML:	0

Understanding representations of uncertainty, an eye-tracking study part II: The effect of expertise

Journal article(s) based on this preprint

Interactive discussion

Interactive discussion

Peer review completion

Suggestions for revision or reasons for rejection

Suggestions for revision or reasons for rejection

Journal article(s) based on this preprint

Viewed

Viewed (geographical distribution)

Cited

1 citations as recorded by crossref.