Fractions Skill Score across spatial scales: statistical significance and discrimination

Abstract. The Fractions Skill Score (FSS) is usually interpreted through its magnitude, by comparison with reference values such as the useful-skill level (1 + f0)/2 or the expected score of a random forecast. Controlled synthetic experiments with summer convective cells, in which the correspondence between forecast and observed cells is known by construction, are used here to separate the magnitude of the score, its statistical significance against forecasts made of independently placed cells, and its ability to discriminate corresponding from unrelated forecasts, for single field pairs and for seasons of 90 aggregated pairs. Three results follow. First, reference values are not significance tests: used as decision rules for one forecast, they are exceeded by 0–99 % of unrelated forecasts depending on width, threshold, domain and number of cells, whereas Monte Carlo critical values keep false positives near 5 %. For aggregated scores, the critical value converges at rate n^−1/2 to the asymptotic limit of the score under the null, and the random-forecast reference of Antonio and Aitchison is this limit for their Bernoulli null when the forecast probability equals the mean observed fraction: a central value, not a percentile. Second, the mean FSS of corresponding forecasts rises steadily with the width, whereas discrimination is largest at a width set by the position error, 5–15 grid points of 2.2 km for cells displaced by up to their radius, and declines beyond it; restricting the windows to the interior of the domain roughly doubles that decline. Third, seasonal aggregation multiplies the separation by about √n, and the most discriminating width of the aggregated score stays at or next to that of a single day: it ranks forecasting systems of different capability, but does not increase the evidence carried by the score of any single forecast. Critical values are specific to the population verified and to the null against which the score is judged; for seasons the delta method gives an asymptotic approximation to them in closed form from the daily sums of the score, without simulation. The seasonal score evaluates a forecasting system; the single-case analysis shows at which scale the score of each of its forecasts carries most evidence.