Big Data Analytics for Earth Sciences: the EarthServer approach

Big Data Analytics is an emerging field since massive storage and computing capabilities have been made available by advanced e-infrastructures. Earth and Environmental sciences are likely to benefit from Big Data Analytics techniques supporting the processing of the large number of Earth Observation datasets currently acquired and generated through observations and simulations. However, Earth Science data and applications present specificities in terms of relevance of the geospatial information, wide heterogeneity of data models and formats, and complexity of processing. Therefore, Big Earth Data Analytics requires specifically tailored techniques and tools. The EarthServer Big Earth Data Analytics engine offers a solution for coverage-type datasets, built around a high performance array database technology, and the adoption and enhancement of standards for service interaction (OGC WCS and WCPS). The EarthServer solution, led by the collection of requirements from scientific communities and international initiatives, provides a holistic approach that ranges from query languages and scalability up to mobile access and visualization. The result is demonstrated and validated through the development of lighthouse applications in the Marine, Geology, Atmospheric, Planetary and Cryospheric science domains.

Big Data Analytics for Earth Sciences: the EarthServer approach

Big Data Analytics is an emerging field since massive storage and computing capabilities have been made available by advanced e-infrastructures. Earth and Environmental sciences are likely to benefit from Big Data Analytics techniques supporting the processing of the large number of Earth Observation datasets currently acquired and generated through observations and simulations. However, Earth Science data and applications present specificities in terms of relevance of the geospatial information, wide heterogeneity of data models and formats, and complexity of processing. Therefore, Big Earth Data Analytics requires specifically tailored techniques and tools. The EarthServer Big Earth Data Analytics engine offers a solution for coverage-type datasets, built around a high performance array database technology, and the adoption and enhancement of standards for service interaction (OGC WCS and WCPS). The EarthServer solution, led by the collection of requirements from scientific communities and international initiatives, provides a holistic approach that ranges from query languages and scalability up to mobile access and visualization. The result is demonstrated and validated through the development of lighthouse applications in the Marine, Geology, Atmospheric, Planetary and Cryospheric science domains.

Warmer, deeper and greener mixed layers in the north Atlantic subpolar gyre over the last 50 years.

Shifts in global climate resonate in plankton dynamics, biogeochemical cycles, and marine food webs. We studied these linkages in the North Atlantic subpolar gyre (NASG), which hosts extensive phytoplankton blooms. We show that phytoplankton abundance increased since the 1960s in parallel to a deepening of the mixed layer and a strengthening of winds and heat losses from the ocean, as driven by the low frequency of the North Atlantic Oscillation (NAO). In parallel to these bottom-up processes, the top-down control of phytoplankton by copepods decreased over the same time period in the western NASG, following sea surface temperature changes typical of the Atlantic Multi-decadal Oscillation (AMO). While previous studies have hypothesized that climate-driven warming would facilitate seasonal stratification of surface waters and long-term phytoplankton increase in subpolar regions, here we show that deeper mixed layers in the NASG can be warmer and host a higher phytoplankton biomass. These results emphasize that different modes of climate variability regulate bottom-up (NAO control) and top-down (AMO control) forcing on phytoplankton at decadal timescales. As a consequence, different relationships between phytoplankton, zooplankton, and their physical environment appear subject to the disparate temporal scale of the observations (seasonal, interannual, or decadal). The prediction of phytoplankton response to climate change should be built upon what is learnt from observations at the longest timescales.

Warmer, deeper and greener mixed layers in the north Atlantic subpolar gyre over the last 50 years.

Shifts in global climate resonate in plankton dynamics, biogeochemical cycles, and marine food webs. We studied these linkages in the North Atlantic subpolar gyre (NASG), which hosts extensive phytoplankton blooms. We show that phytoplankton abundance increased since the 1960s in parallel to a deepening of the mixed layer and a strengthening of winds and heat losses from the ocean, as driven by the low frequency of the North Atlantic Oscillation (NAO). In parallel to these bottom-up processes, the top-down control of phytoplankton by copepods decreased over the same time period in the western NASG, following sea surface temperature changes typical of the Atlantic Multi-decadal Oscillation (AMO). While previous studies have hypothesized that climate-driven warming would facilitate seasonal stratification of surface waters and long-term phytoplankton increase in subpolar regions, here we show that deeper mixed layers in the NASG can be warmer and host a higher phytoplankton biomass. These results emphasize that different modes of climate variability regulate bottom-up (NAO control) and top-down (AMO control) forcing on phytoplankton at decadal timescales. As a consequence, different relationships between phytoplankton, zooplankton, and their physical environment appear subject to the disparate temporal scale of the observations (seasonal, interannual, or decadal). The prediction of phytoplankton response to climate change should be built upon what is learnt from observations at the longest timescales.

Near-real-time service provision during effusive crises at Etna and Stromboli: basis and implementation of satellite-based IR operations

Using the NEODAAS-Dundee AVHRR receiving station (Scotland), NEODAAS-Plymouth can provide calibrated brightness temperature data to end users or interim users in near-real time. Between 2000 and 2009 these data were used to undertake volcano hot spot detection, reporting and time-average discharge rate dissemination during effusive crises at Mount Etna and Stromboli (Italy). Data were passed via FTP, within an hour of image generation, to the hot spot detection system maintained at Hawaii Institute of Geophysics and Planetology (HIGP, University of Hawaii at Manoa, Honolulu, USA). Final product generation and quality control were completed manually at HIGP once a day, so as to provide information to onsite monitoring agencies for their incorporation into daily reporting duties to Italian Civil Protection. We here describe the processing and dissemination chain, which was designed so as to provide timely, useable, quality-controlled and relevant information for ‘one voice’ reporting by the responsible monitoring agencies.

Near-real-time service provision during effusive crises at Etna and Stromboli: basis and implementation of satellite-based IR operations

Using the NEODAAS-Dundee AVHRR receiving station (Scotland), NEODAAS-Plymouth can provide calibrated brightness temperature data to end users or interim users in near-real time. Between 2000 and 2009 these data were used to undertake volcano hot spot detection, reporting and time-average discharge rate dissemination during effusive crises at Mount Etna and Stromboli (Italy). Data were passed via FTP, within an hour of image generation, to the hot spot detection system maintained at Hawaii Institute of Geophysics and Planetology (HIGP, University of Hawaii at Manoa, Honolulu, USA). Final product generation and quality control were completed manually at HIGP once a day, so as to provide information to onsite monitoring agencies for their incorporation into daily reporting duties to Italian Civil Protection. We here describe the processing and dissemination chain, which was designed so as to provide timely, useable, quality-controlled and relevant information for ‘one voice’ reporting by the responsible monitoring agencies.

A reconciliation of empirical and mechanistic models of the air-sea gas transfer velocity

Models of the air-sea transfer velocity of gases may be either empirical or mechanistic. Extrapolations of empirical models to an unmeasured gas or to another water temperature can be erroneous if the basis of that extrapolation is flawed. This issue is readily demonstrated for the most well-known empirical gas transfer velocity models where the influence of bubble-mediated transfer, which can vary between gases, is not explicitly accounted for. Mechanistic models are hindered by an incomplete knowledge of the mechanisms of air-sea gas transfer. We describe a hybrid model that incorporates a simple mechanistic view—strictly enforcing a distinction between direct and bubble-mediated transfer—but also uses parameterizations based on data from eddy flux measurements of dimethyl sulphide (DMS) to calibrate the model together with dual tracer results to evaluate the model. This model underpins simple algorithms that can be easily applied within schemes to calculate local, regional, or global air-sea fluxes of gases.

A reconciliation of empirical and mechanistic models of the air-sea gas transfer velocity

Models of the air-sea transfer velocity of gases may be either empirical or mechanistic. Extrapolations of empirical models to an unmeasured gas or to another water temperature can be erroneous if the basis of that extrapolation is flawed. This issue is readily demonstrated for the most well-known empirical gas transfer velocity models where the influence of bubble-mediated transfer, which can vary between gases, is not explicitly accounted for. Mechanistic models are hindered by an incomplete knowledge of the mechanisms of air-sea gas transfer. We describe a hybrid model that incorporates a simple mechanistic view—strictly enforcing a distinction between direct and bubble-mediated transfer—but also uses parameterizations based on data from eddy flux measurements of dimethyl sulphide (DMS) to calibrate the model together with dual tracer results to evaluate the model. This model underpins simple algorithms that can be easily applied within schemes to calculate local, regional, or global air-sea fluxes of gases.

Simple heterogeneity parametrization for sea surface temperature and chlorophyll

Using satellite maps this paper offers a complex analysis of chlorophyll & SST heterogeneity in the shelf seas around the southwest of the UK. The heterogeneity scaling follows a simple power law and is consequently pa- rametrized by two parameters. It is shown that in most cases these two parameters vary only relatively little with time. The paper offers a detailed comparison of field heterogeneity between different regions. How much hetero- geneity is in each region preserved in the annual median data is also determined. The paper explicitly demon- strates how one can use these results to calculate representative measurement area for in situ networks.

Simple heterogeneity parametrization for sea surface temperature and chlorophyll

Using satellite maps this paper offers a complex analysis of chlorophyll & SST heterogeneity in the shelf seas around the southwest of the UK. The heterogeneity scaling follows a simple power law and is consequently pa- rametrized by two parameters. It is shown that in most cases these two parameters vary only relatively little with time. The paper offers a detailed comparison of field heterogeneity between different regions. How much hetero- geneity is in each region preserved in the annual median data is also determined. The paper explicitly demon- strates how one can use these results to calculate representative measurement area for in situ networks.

Underway spectrophotometry along the Atlantic Meridional Transect reveals high performance in satellite chlorophyll retrievals

To evaluate the performance of ocean-colour retrievals of total chlorophyll-a concentration requires direct comparison with concomitant and co-located in situ data. For global comparisons, these in situ match-ups should be ideally representative of the distribution of total chlorophyll-a concentration in the global ocean. The oligotrophic gyres constitute the majority of oceanic water, yet are under-sampled due to their inaccessibility and under-represented in global in situ databases. The Atlantic Meridional Transect (AMT) is one of only a few programmes that consistently sample oligotrophic waters. In this paper, we used a spectrophotometer on two AMT cruises (AMT19 and AMT22) to continuously measure absorption by particles in the water of the ship's flow-through system. From these optical data continuous total chlorophyll-a concentrations were estimated with high precision and accuracy along each cruise and used to evaluate the performance of ocean-colour algorithms. We conducted the evaluation using level 3 binned ocean-colour products, and used the high spatial and temporal resolution of the underway system to maximise the number of match-ups on each cruise. Statistical comparisons show a significant improvement in the performance of satellite chlorophyll algorithms over previous studies, with root mean square errors on average less than half (~ 0.16 in log10 space) that reported previously using global datasets (~ 0.34 in log10 space). This improved performance is likely due to the use of continuous absorption-based chlorophyll estimates, that are highly accurate, sample spatial scales more comparable with satellite pixels, and minimise human errors. Previous comparisons might have reported higher errors due to regional biases in datasets and methodological inconsistencies between investigators. Furthermore, our comparison showed an underestimate in satellite chlorophyll at low concentrations in 2012 (AMT22), likely due to a small bias in satellite remote-sensing reflectance data. Our results highlight the benefits of using underway spectrophotometric systems for evaluating satellite ocean-colour data and underline the importance of maintaining in situ observatories that sample the oligotrophic gyres.

Underway spectrophotometry along the Atlantic Meridional Transect reveals high performance in satellite chlorophyll retrievals

To evaluate the performance of ocean-colour retrievals of total chlorophyll-a concentration requires direct comparison with concomitant and co-located in situ data. For global comparisons, these in situ match-ups should be ideally representative of the distribution of total chlorophyll-a concentration in the global ocean. The oligotrophic gyres constitute the majority of oceanic water, yet are under-sampled due to their inaccessibility and under-represented in global in situ databases. The Atlantic Meridional Transect (AMT) is one of only a few programmes that consistently sample oligotrophic waters. In this paper, we used a spectrophotometer on two AMT cruises (AMT19 and AMT22) to continuously measure absorption by particles in the water of the ship's flow-through system. From these optical data continuous total chlorophyll-a concentrations were estimated with high precision and accuracy along each cruise and used to evaluate the performance of ocean-colour algorithms. We conducted the evaluation using level 3 binned ocean-colour products, and used the high spatial and temporal resolution of the underway system to maximise the number of match-ups on each cruise. Statistical comparisons show a significant improvement in the performance of satellite chlorophyll algorithms over previous studies, with root mean square errors on average less than half (~ 0.16 in log10 space) that reported previously using global datasets (~ 0.34 in log10 space). This improved performance is likely due to the use of continuous absorption-based chlorophyll estimates, that are highly accurate, sample spatial scales more comparable with satellite pixels, and minimise human errors. Previous comparisons might have reported higher errors due to regional biases in datasets and methodological inconsistencies between investigators. Furthermore, our comparison showed an underestimate in satellite chlorophyll at low concentrations in 2012 (AMT22), likely due to a small bias in satellite remote-sensing reflectance data. Our results highlight the benefits of using underway spectrophotometric systems for evaluating satellite ocean-colour data and underline the importance of maintaining in situ observatories that sample the oligotrophic gyres.

Cross-calibrating ALES Envisat and CryoSat-2 Delay-Doppler: A coastal altimetry study in the Indonesian Seas

A regional cross-calibration between the first Delay Doppler altimetry dataset from Cryosat-2 and a retracked Envisat dataset is here presented, in order to test the benefits of the Delay-Doppler processing and to expand the Envisat time series in the coastal ocean. The Indonesian Seas are chosen for the calibration, since the availability of altimetry data in this region is particularly beneficial due to the lack of in-situ measurements and its importance for global ocean circulation. The Envisat data in the region are retracked with the Adaptive Leading Edge Subwaveform (ALES) Retracker, which has been previously validated and applied successfully to coastal sea level research. The study demonstrates that CryoSat-2 is able to decrease the 1-Hz noise of sea level estimations by 0.3 cm within 50 km of the coast, when compared to the ALES-reprocessed Envisat dataset. It also shows that Envisat can be confidently used for detailed oceanographic research after the orbit change of October 2010. Cross-calibration at the crossover points indicates that in the region of study a sea state bias correction equal to 5% of the significant wave height is an acceptable approximation for Delay-Doppler altimetry. The analysis of the joint sea level time series reveals the geographic extent of the semiannual signal caused by Kelvin waves during the monsoon transitions, the larger amplitudes of the annual signal due to the Java Coastal Current and the impact of the strong La Nina event of 2010 on rising sea level trends.

Cross-calibrating ALES Envisat and CryoSat-2 Delay-Doppler: A coastal altimetry study in the Indonesian Seas

A regional cross-calibration between the first Delay Doppler altimetry dataset from Cryosat-2 and a retracked Envisat dataset is here presented, in order to test the benefits of the Delay-Doppler processing and to expand the Envisat time series in the coastal ocean. The Indonesian Seas are chosen for the calibration, since the availability of altimetry data in this region is particularly beneficial due to the lack of in-situ measurements and its importance for global ocean circulation. The Envisat data in the region are retracked with the Adaptive Leading Edge Subwaveform (ALES) Retracker, which has been previously validated and applied successfully to coastal sea level research. The study demonstrates that CryoSat-2 is able to decrease the 1-Hz noise of sea level estimations by 0.3 cm within 50 km of the coast, when compared to the ALES-reprocessed Envisat dataset. It also shows that Envisat can be confidently used for detailed oceanographic research after the orbit change of October 2010. Cross-calibration at the crossover points indicates that in the region of study a sea state bias correction equal to 5% of the significant wave height is an acceptable approximation for Delay-Doppler altimetry. The analysis of the joint sea level time series reveals the geographic extent of the semiannual signal caused by Kelvin waves during the monsoon transitions, the larger amplitudes of the annual signal due to the Java Coastal Current and the impact of the strong La Nina event of 2010 on rising sea level trends.

Using multifractals to evaluate oceanographic model skill

We are in an era of unprecedented data volumes generated from observations and model simulations. This is particularly true from satellite Earth Observations (EO) and global scale oceanographic models. This presents us with an opportunity to evaluate large scale oceanographic model outputs using EO data. Previous work on model skill evaluation has led to a plethora of metrics. The paper defines two new model skill evaluation metrics. The metrics are based on the theory of universal multifractals and their purpose is to measure the structural similarity between the model predictions and the EO data. The two metrics have the following advantages over the standard techniques: a) they are scale-free, b) they carry important part of information about how model represents different oceanographic drivers. Those two metrics are then used in the paper to evaluate the performance of the FVCOM model in the shelf seas around the south-west coast of the UK.