A distributed architecture for the monitoring and analysis of time series data

It is estimated that the quantity of digital data being transferred, processed or stored at any one time currently stands at 4.4 zettabytes (4.4 × 2 70 bytes) and this figure is expected to have grown by a factor of 10 to 44 zettabytes by 2020. Exploiting this data is, and will remain, a significant challenge. At present there is the capacity to store 33% of digital data in existence at any one time; by 2020 this capacity is expected to fall to 15%. These statistics suggest that, in the era of Big Data, the identification of important, exploitable data will need to be done in a timely manner. Systems for the monitoring and analysis of data, e.g. stock markets, smart grids and sensor networks, can be made up of massive numbers of individual components. These components can be geographically distributed yet may interact with one another via continuous data streams, which in turn may affect the state of the sender or receiver. This introduces a dynamic causality, which further complicates the overall system by introducing a temporal constraint that is difficult to accommodate. Practical approaches to realising the system described above have led to a multiplicity of analysis techniques, each of which concentrates on specific characteristics of the system being analysed and treats these characteristics as the dominant component affecting the results being sought. The multiplicity of analysis techniques introduces another layer of heterogeneity, that is heterogeneity of approach, partitioning the field to the extent that results from one domain are difficult to exploit in another. The question is asked can a generic solution for the monitoring and analysis of data that: accommodates temporal constraints; bridges the gap between expert knowledge and raw data; and enables data to be effectively interpreted and exploited in a transparent manner, be identified? The approach proposed in this dissertation acquires, analyses and processes data in a manner that is free of the constraints of any particular analysis technique, while at the same time facilitating these techniques where appropriate. Constraints are applied by defining a workflow based on the production, interpretation and consumption of data. This supports the application of different analysis techniques on the same raw data without the danger of incorporating hidden bias that may exist. To illustrate and to realise this approach a software platform has been created that allows for the transparent analysis of data, combining analysis techniques with a maintainable record of provenance so that independent third party analysis can be applied to verify any derived conclusions. In order to demonstrate these concepts, a complex real world example involving the near real-time capturing and analysis of neurophysiological data from a neonatal intensive care unit (NICU) was chosen. A system was engineered to gather raw data, analyse that data using different analysis techniques, uncover information, incorporate that information into the system and curate the evolution of the discovered knowledge. The application domain was chosen for three reasons: firstly because it is complex and no comprehensive solution exists; secondly, it requires tight interaction with domain experts, thus requiring the handling of subjective knowledge and inference; and thirdly, given the dearth of neurophysiologists, there is a real world need to provide a solution for this domain

Evaluating the spatial transferability and temporal repeatability of remote-sensing-based lake water quality retrieval algorithms at the European scale: a meta-analysis approach

Many studies have shown the considerable potential for the application of remote-sensing-based methods for deriving estimates of lake water quality. However, the reliable application of these methods across time and space is complicated by the diversity of lake types, sensor configuration, and the multitude of different algorithms proposed. This study tested one operational and 46 empirical algorithms sourced from the peer-reviewed literature that have individually shown potential for estimating lake water quality properties in the form of chlorophyll-a (algal biomass) and Secchi disc depth (SDD) (water transparency) in independent studies. Nearly half (19) of the algorithms were unsuitable for use with the remote-sensing data available for this study. The remaining 28 were assessed using the Terra/Aqua satellite archive to identify the best performing algorithms in terms of accuracy and transferability within the period 2001–2004 in four test lakes, namely Vänern, Vättern, Geneva, and Balaton. These lakes represent the broad continuum of large European lake types, varying in terms of eco-region (latitude/longitude and altitude), morphology, mixing regime, and trophic status. All algorithms were tested for each lake separately and combined to assess the degree of their applicability in ecologically different sites. None of the algorithms assessed in this study exhibited promise when all four lakes were combined into a single data set and most algorithms performed poorly even for specific lake types. A chlorophyll-a retrieval algorithm originally developed for eutrophic lakes showed the most promising results (R2 = 0.59) in oligotrophic lakes. Two SDD retrieval algorithms, one originally developed for turbid lakes and the other for lakes with various characteristics, exhibited promising results in relatively less turbid lakes (R2 = 0.62 and 0.76, respectively). The results presented here highlight the complexity associated with remotely sensed lake water quality estimates and the high degree of uncertainty due to various limitations, including the lake water optical properties and the choice of methods.

Soft sediment ecology of Lough Hyne Marine Reserve - spatial and temporal patterns

Distribution of soft sediment benthic fauna and the environmental factors affecting them were studied, to investigate changes across spatial and temporal scales. Investigations took place at Lough Hyne Marine Reserve using a range of methods. Data on the sedimentation rates of organic and inorganic matter were collected at monthly intervals for one year at a number of sites around the Lough, by use of vertical midwater-column sediment traps. Sedimentation of these two fractions were not coupled; inorganic matter sedimentation depended on hydrodynamic and weather factors, while the organic matter sedimentation was more complex, being dependent on biological and chemical processes in the water column. The effects of regular hypoxic episodes on benthic fauna due to a natural seasonal thermocline were studied in the deep Western Trough, using camera-equipped remotely-operated vehicle to follow transects, on a three-monthly basis over one year. In late summer, the area below the thermocline of the Western Trough was devoid of visible fauna. Decapod crustaceans were the first taxon to make use of ameliorating oxygen conditions in autumn, by darting below the thermocline depth, most likely to scavenge. This was indicated by tracks that they left on the surface of the Trough floor. Some species, most noticeably Fries’ goby Lesueurigobius friesii, migrated below the thermocline depth when conditions were normoxic and established semi-permanent burrows. Their population encompassed all size classes, indicating that this habitat was not limited to juveniles of this territorial species. Recolonisation by macrofauna and burrowing megafauna was studied during normoxic conditions, from November 2009 to May 2010. Macrofauna displayed a typical post-disturbance pattern of recolonisation with one species, the polychaete Scalibregma inflatum, occurring at high abundance levels in March 2010. In May, this population had become significantly reduced and a more diverse community was established. The abundance of burrowing infauna comprising decapods crabs and Fries’ gobies, was estimated by identifying and counting their distinctive burrow structures. While above the summer thermocline depth, burrow abundance increased in a linear fashion, below the thermocline depth a slight reduction of burrow abundance occurred in May, when oxygen conditions deteriorated again. The majority of the burrows occurring in May were made by Fries’ gobies, which are thought to encounter low oxygen concentrations in their burrows. Reduction in burrow abundance of burrowing shrimps Calocaris macandreae and Callianassa subterranea (based on descriptions of burrow structures from the literature), from March to May, might be related to their reduced activity in hypoxia, leading to loss of structural burrow maintenance. Spatial and temporal changes to macrofaunal assemblage structures were studied seasonally for one year across 5 sites in the Lough and subject to multivariate statistical analysis. Assemblage structures were significantly correlated with organic matter levels in the sediment, the amounts of organic matter settling out of the water column one month before macrofaunal sampling took place as well as current speed and temperature. This study was the first to investigate patterns and processes in the Lough soft sediment ecology across all 3 basins on a temporal and spatial scale. An investigation into the oceanographic aspects of the development, behaviour and break-down of the summer thermocline of Lough Hyne was performed in collaboration with researchers from other Irish institutions.

Variability in north temperate dung beetle assemblages at different spatial and temporal scales

This study examined the spatial and temporal variability of dung beetle assemblages across a variety of scales e.g. from the between-pad scale (examining the effects of dung size and type) to larger spatial scales encompassing southern Ireland. Dung beetle assemblage structure as sampled by dung pad cohort samples and dung baited pitfall trapping were compared. Generally, the rank order of abundance of dung beetle species was significantly correlated between pitfall catches and cohort pad samples. Across different dung sizes, in both pitfall catches and cohort pad samples, the relative abundance of species was frequently significantly different, but the rank order of abundance of dung beetle was usually significantly correlated. Considerable variations in pitfall catches at temporal scales of a few days appeared to be closely related to weather conditions and rotational grazing. However, despite considerable variation in absolute abundances between consecutive days of sampling, assemblage structure typically remained very similar. The relationship between dung pad size and dung beetle colonisation was investigated. In field experiments in which pads of different sizes (0.25 L, 0.5 L, 1.0 L and 1.5 L) were artificially deposited, there was a positive relationship between pad size and both biomass and number of beetles colonising dung pads and pitfall traps. In addition, with one exception, the field experiments indicated a general positive relationship between dung pad size and biomass density (dung beetle biomass per unit dung volume). A laboratory experiment indicated that pat residence times of A. rufipes were significantly correlated with dung pad size. Investigation of naturally-deposited cow dung pads in the field also indicated that both larval numbers and densities were significantly correlated with dung pad size. These results were discussed in the context of theory related to aggregation and coexistence of species, and resource utilisation by organisms in ephemeral, patchy resources. The colonisation by dung beetles of dung types from native herbivores (sheep, horse and cow) was investigated in field experiments. There were significant differences between the dung types in the chemical parameters measured, and there were significant differences in abundances of dung beetles colonising the dung types. Sheep dung was typically the preferred dung type. Data from these field experiments, and from published literature, indicated that dung beetle species can display dung type preferences, in terms of comparisons of both absolute and relative abundances. In addition, data from laboratory experiments indicate that both Aphodius larval production and pat residence times tended to be higher in those dung types which were preferred by adult Aphodius in the colonisation experiments. Data from dung-baited pitfall trapping (from this and another study) at several sites (up to 180 km distant) and over a number of years (between 1991 and 1996) were used to investigate spatial and temporal variation in dung beetle assemblage structure and composition (Aphodius, Sphaeridium and Geotrupes) across a range of scales in southern Ireland. Species richness levels, species composition and rank order of abundances were very similar between the assemblages. The temporal variability between seasons within any year exceeded temporal variability between years. DCA ordinations indicated that there was a similar level of variability between assemblage structure from the between-field (~1km) to regional (~180 km) spatial scales, and between year (6 years) temporal scales. At the biogeographical spatial scale, analysis of data from the literature indicated that there was considerable variability at this scale, largely due to species turnover.

Reasoning with sorted-pareto dominance and other qualitative and partially ordered preferences in soft constraints

In decision making problems where we need to choose a particular decision or alternative from a set of possible choices, we often have some preferences which determine if we prefer one decision over another. When these preferences give us an ordering on the decisions that is complete, then it is easy to choose the best or one of the best decisions. However it often occurs that the preferences relation is partially ordered, and we have no best decision. In this thesis, we look at what happens when we have such a partial order over a set of decisions, in particular when we have multiple orderings on a set of decisions, and we present a framework for qualitative decision making. We look at the different natural notions of optimal decision that occur in this framework, which gives us different optimality classes, and we examine the relationships between these classes. We then look in particular at a qualitative preference relation called Sorted-Pareto Dominance, which is an extension of Pareto Dominance, and we give a semantics for this relation as one that is compatible with any order-preserving mapping of an ordinal preference scale to a numerical one. We apply Sorted-Pareto dominance to a Soft Constraints setting, where we solve problems in which the soft constraints associate qualitative preferences to decisions in a decision problem. We also examine the Sorted-Pareto dominance relation in the context of our qualitative decision making framework, looking at the relevant optimality classes for the Sorted-Pareto case, which gives us classes of decisions that are necessarily optimal, and optimal for some choice of mapping of an ordinal scale to a quantitative one. We provide some empirical analysis of Sorted-Pareto constraints problems and examine the optimality classes that result.