Employing smoothness of the time series of sky radiances measured in the solar aureole for cloud screening

Abstract. Cloud screening algorithms have always been a critical component of Aerosol Robotic Network (AERONET) aerosol optical depth (AOD) Level 1.5 and 2.0 product. The initial cloud screening algorithm in the Version 1 and 2 database was semi-automatic and required involvement of human analyst to finalize  the results. It became fully automatic in Version 3 (V3) due to  employing information on the angular shape of sky radiances measured in aureole (curvature algorithm). Although efficient, the curvature algorithm is threshold based and fails to detect clouds when its parameters are beyond the corresponding pre-determined thresholds. This is especially noticeable at high latitudes where the size of ice crystals in cirrus clouds are sometimes relatively small and therefore comparable in size to aerosols. It is shown that additional information can be extracted from analysis of the smoothness of diurnal variability of sky radiances measured at the 3.3-degree scattering angle. This measurement is a part of  so-called curvature scan (CCS), which takes  measurements from 3 to 7.5 degrees scattering angle with 0.3-degree steps after each measurement of AOD. The analysis of the diurnal variability of CCS (3.3) for cloud-free conditions shows relatively smooth temporal dependencies, which can be fitted by polynomials with high correlation coefficients while in conditions almost completely dominated by clouds, the temporal variability is completely random. For partially cloudy days, the two main features are observed: relatively smooth aerosol signature and irregular spikes due to clouds. The new technique is proposed that employs the smoothness of the diurnal variability of CCS(3.3) scan as a criterion of the cloud free conditions. In the case when both features are present, the idea of the new algorithm is to remove irregular spikes due to clouds while  keeping smooth part due to aerosols intact. The new algorithm detects spikes associated with clouds by comparing magnitudes of CCS(3.3) at neighboring time stamps through calculating their first differences (FD). This algorithm was applied to the CCS(3.3) measurements taken at several AERONET sites. The results were analyzed in terms of net change in Angstrom exponent (AE) as well as number of AOD measurements. The analysis showed the algorithm performs satisfactorily  at AERONET sites dominated by fine mode aerosols, however at sites dominated by dust, the algorithm removes a big fraction of cloud-free observations. The issue was corrected by introducing an additional cloud screening parameter. It is based on observation of the different rate in changing  of AE with iterations for cloud-free and cloudy conditions with much higher rate in the former case. The new parameter was selected as a slope of the linear regression between integration number and the value of AE after the corresponding iteration. Algorithm disregards FD algorithm results if the slope is smaller than certain threshold value. Finalizing  the FD algorithm threshold setting as well as evaluation of the algorithm performance is done by using independent cloud detection information available from Micro-Pulse Lidar Network (MPLNET) data. The AERONET and MPLNET data were time and space collocated with additional averaging over one hour period. The comparison showed that, on average, the FD algorithm outperformed V3 L1.5 by about 0.02 in Mathews Correlation Coefficient (MCC), suggesting consistent improvement in overall cloud detection accuracy. Additional analysis performed in terms of MCC metrics also showed  that the FD algorithm achieves a more balanced and accurate classification of clouds vs clear.