SPARE-ICE: synergistic Ice Water Path from passive operational sensors

This article presents SPARE-ICE, the Synergistic Passive Atmospheric Retrieval Experiment-ICE. SPARE-ICE is the first Ice Water Path (IWP) product combining infrared and microwave radiances. By using only passive operational sensors, the SPARE-ICE retrieval can be used to process data from at least the NOAA 15 to 19 and MetOp satellites, obtaining time series from 1998 onward. The retrieval is developed using collocations between passive operational sensors (solar, terrestrial infrared, microwave), the CloudSat radar, and the CALIPSO lidar. The collocations form a retrieval database matching measurements from passive sensors against the existing active combined radar-lidar product 2C-ICE. With this retrieval database, we train a pair of artificial neural networks to detect clouds and retrieve IWP. When considering solar, terrestrial infrared, and microwave-based measurements, we show that any combination of two techniques performs better than either single-technique retrieval. We choose not to include solar reflectances in SPARE-ICE, because the improvement is small, and so that SPARE-ICE can be retrieved both daytime and nighttime. The median fractional error between SPARE-ICE and 2C-ICE is around a factor 2, a figure similar to the random error between 2C-ICE ice water content (IWC) and in situ measurements. A comparison of SPARE-ICE with Moderate Resolution Imaging Spectroradiometer (MODIS), Pathfinder Atmospheric Extended (PATMOS-X), and Microwave Surface and Precipitation Products System (MSPPS) indicates that SPARE-ICE appears to perform well even in difficult conditions. SPARE-ICE is available for public use.

An approximate dynamic programming approach for improving accuracy of lossy data compression by Bloom filters

Bloom filters are a data structure for storing data in a compressed form. They offer excellent space and time efficiency at the cost of some loss of accuracy (so-called lossy compression). This work presents a yes-no Bloom filter, which as a data structure consisting of two parts: the yes-filter which is a standard Bloom filter and the no-filter which is another Bloom filter whose purpose is to represent those objects that were recognised incorrectly by the yes-filter (that is, to recognise the false positives of the yes-filter). By querying the no-filter after an object has been recognised by the yes-filter, we get a chance of rejecting it, which improves the accuracy of data recognition in comparison with the standard Bloom filter of the same total length. A further increase in accuracy is possible if one chooses objects to include in the no-filter so that the no-filter recognises as many as possible false positives but no true positives, thus producing the most accurate yes-no Bloom filter among all yes-no Bloom filters. This paper studies how optimization techniques can be used to maximize the number of false positives recognised by the no-filter, with the constraint being that it should recognise no true positives. To achieve this aim, an Integer Linear Program (ILP) is proposed for the optimal selection of false positives. In practice the problem size is normally large leading to intractable optimal solution. Considering the similarity of the ILP with the Multidimensional Knapsack Problem, an Approximate Dynamic Programming (ADP) model is developed making use of a reduced ILP for the value function approximation. Numerical results show the ADP model works best comparing with a number of heuristics as well as the CPLEX built-in solver (B&B), and this is what can be recommended for use in yes-no Bloom filters. In a wider context of the study of lossy compression algorithms, our researchis an example showing how the arsenal of optimization methods can be applied to improving the accuracy of compressed data.

The role of big data in governance: a regulatory and legal perspective of analytics in global financial services

This paper discusses how global financial institutions are using big data analytics within their compliance operations. A lot of previous research has focused on the strategic implications of big data, but not much research has considered how such tools are entwined with regulatory breaches and investigations in financial services. Our work covers two in-depth qualitative case studies, each addressing a distinct type of analytics. The first case focuses on analytics which manage everyday compliance breaches and so are expected by managers. The second case focuses on analytics which facilitate investigation and litigation where serious unexpected breaches may have occurred. In doing so, the study focuses on the micro/data to understand how these tools are influencing operational risks and practices. The paper draws from two bodies of literature, the social studies of information systems and finance to guide our analysis and practitioner recommendations. The cases illustrate how technologies are implicated in multijurisdictional challenges and regulatory conflicts at each end of the operational risk spectrum. We find that compliance analytics are both shaping and reporting regulatory matters yet often firms may have difficulties in recruiting individuals with relevant but diverse skill sets. The cases also underscore the increasing need for financial organizations to adopt robust information governance policies and processes to ease future remediation efforts.