Optimizing the reliability and resource efficiency of MapReduce-based systems

Debido al gran incremento de datos digitales que ha tenido lugar en los últimos años, ha surgido un nuevo paradigma de computación paralela para el procesamiento eficiente de grandes volúmenes de datos. Muchos de los sistemas basados en este paradigma, también llamados sistemas de computación intensiva de datos, siguen el modelo de programación de Google MapReduce. La principal ventaja de los sistemas MapReduce es que se basan en la idea de enviar la computación donde residen los datos, tratando de proporcionar escalabilidad y eficiencia. En escenarios libres de fallo, estos sistemas generalmente logran buenos resultados. Sin embargo, la mayoría de escenarios donde se utilizan, se caracterizan por la existencia de fallos. Por tanto, estas plataformas suelen incorporar características de tolerancia a fallos y fiabilidad. Por otro lado, es reconocido que las mejoras en confiabilidad vienen asociadas a costes adicionales en recursos. Esto es razonable y los proveedores que ofrecen este tipo de infraestructuras son conscientes de ello. No obstante, no todos los enfoques proporcionan la misma solución de compromiso entre las capacidades de tolerancia a fallo (o de manera general, las capacidades de fiabilidad) y su coste. Esta tesis ha tratado la problemática de la coexistencia entre fiabilidad y eficiencia de los recursos en los sistemas basados en el paradigma MapReduce, a través de metodologías que introducen el mínimo coste, garantizando un nivel adecuado de fiabilidad. Para lograr esto, se ha propuesto: (i) la formalización de una abstracción de detección de fallos; (ii) una solución alternativa a los puntos únicos de fallo de estas plataformas, y, finalmente, (iii) un nuevo sistema de asignación de recursos basado en retroalimentación a nivel de contenedores. Estas contribuciones genéricas han sido evaluadas tomando como referencia la arquitectura Hadoop YARN, que, hoy en día, es la plataforma de referencia en la comunidad de los sistemas de computación intensiva de datos. En la tesis se demuestra cómo todas las contribuciones de la misma superan a Hadoop YARN tanto en fiabilidad como en eficiencia de los recursos utilizados. ABSTRACT Due to the increase of huge data volumes, a new parallel computing paradigm to process big data in an efficient way has arisen. Many of these systems, called dataintensive computing systems, follow the Google MapReduce programming model. The main advantage of these systems is based on the idea of sending the computation where the data resides, trying to provide scalability and efficiency. In failure-free scenarios, these frameworks usually achieve good results. However, these ones are not realistic scenarios. Consequently, these frameworks exhibit some fault tolerance and dependability techniques as built-in features. On the other hand, dependability improvements are known to imply additional resource costs. This is reasonable and providers offering these infrastructures are aware of this. Nevertheless, not all the approaches provide the same tradeoff between fault tolerant capabilities (or more generally, reliability capabilities) and cost. In this thesis, we have addressed the coexistence between reliability and resource efficiency in MapReduce-based systems, looking for methodologies that introduce the minimal cost and guarantee an appropriate level of reliability. In order to achieve this, we have proposed: (i) a formalization of a failure detector abstraction; (ii) an alternative solution to single points of failure of these frameworks, and finally (iii) a novel feedback-based resource allocation system at the container level. Finally, our generic contributions have been instantiated for the Hadoop YARN architecture, which is the state-of-the-art framework in the data-intensive computing systems community nowadays. The thesis demonstrates how all our approaches outperform Hadoop YARN in terms of reliability and resource efficiency.

The charter of the Albany and Greenbush Bridge Company : and a statement of its claims on the public interest in connection with the Boston, Hoosac Tunnel and Albany Railroad.

"J.W. Gascoigne, secretary."

Prison life of Jefferson Davis : embracing details and incidents in his captivity, particulars concerning his health and habits, together with many conversations on topics of great public interest /

In the Historical magazine for March, 1869, this work is erroneously ascribed to Gen. Halpine.

The school administrator and his publications; building public interest in education,

Mode of access: Internet.

Modern methods of charity; an account of the systems of relief, public and private, in the principal countries having modern methods,

Bibliography: p. 689-702.

Proposed Passenger train act of 1960. Hearings before the Surface Transportation Subcommittee of the Committee on Interstate and Foreign Commerce, United States Senate, Eighty-sixth Congress, second session, on S. 3020, a bill to amend the Interstate commerce act, as amended, so as to strengthen and improve the national transportation system, insure the protection of the public interest, and for other purposes ...

Mode of access: Internet.

The riddle of the Rhine; chemical strategy in peace and war. An account of the critical struggle for power and for the decisive war initiative. The campaign fostered by the great Rhine factories, and the pressing problems which they represent. A matter of pre-eminent public interest concerning the sincerity of disarmament, the future of warfare, and the stability of peace.

"The existence of this giant monopoly [Interessen gemeinschaft der deutschen teerfarbenfabriken] raises vital military and economic questions, which are ... 'The riddle of the Rhine'."--p. 14.

Dynamic QoS management and optimisation in service-based systems

Service-based systems that are dynamically composed at run time to provide complex, adaptive functionality are currently one of the main development paradigms in software engineering. However, the Quality of Service (QoS) delivered by these systems remains an important concern, and needs to be managed in an equally adaptive and predictable way. To address this need, we introduce a novel, tool-supported framework for the development of adaptive service-based systems called QoSMOS (QoS Management and Optimisation of Service-based systems). QoSMOS can be used to develop service-based systems that achieve their QoS requirements through dynamically adapting to changes in the system state, environment and workload. QoSMOS service-based systems translate high-level QoS requirements specified by their administrators into probabilistic temporal logic formulae, which are then formally and automatically analysed to identify and enforce optimal system configurations. The QoSMOS self-adaptation mechanism can handle reliability- and performance-related QoS requirements, and can be integrated into newly developed solutions or legacy systems. The effectiveness and scalability of the approach are validated using simulations and a set of experiments based on an implementation of an adaptive service-based system for remote medical assistance.

Knowledge-based systems and engineering software

The work described was carried out as part of a collaborative Alvey software engineering project (project number SE057). The project collaborators were the Inter-Disciplinary Higher Degrees Scheme of the University of Aston in Birmingham, BIS Applied Systems Ltd. (BIS) and the British Steel Corporation. The aim of the project was to investigate the potential application of knowledge-based systems (KBSs) to the design of commercial data processing (DP) systems. The work was primarily concerned with BIS's Structured Systems Design (SSD) methodology for DP systems development and how users of this methodology could be supported using KBS tools. The problems encountered by users of SSD are discussed and potential forms of computer-based support for inexpert designers are identified. The architecture for a support environment for SSD is proposed based on the integration of KBS and non-KBS tools for individual design tasks within SSD - The Intellipse system. The Intellipse system has two modes of operation - Advisor and Designer. The design, implementation and user-evaluation of Advisor are discussed. The results of a Designer feasibility study, the aim of which was to analyse major design tasks in SSD to assess their suitability for KBS support, are reported. The potential role of KBS tools in the domain of database design is discussed. The project involved extensive knowledge engineering sessions with expert DP systems designers. Some practical lessons in relation to KBS development are derived from this experience. The nature of the expertise possessed by expert designers is discussed. The need for operational KBSs to be built to the same standards as other commercial and industrial software is identified. A comparison between current KBS and conventional DP systems development is made. On the basis of this analysis, a structured development method for KBSs in proposed - the POLITE model. Some initial results of applying this method to KBS development are discussed. Several areas for further research and development are identified.

Nonlinear spectral shaping and optical rogue events in fiber-based systems

We provide an overview of our recent work on the shaping and stability of optical continua in the long pulse regime. Fibers with normal group-velocity dispersion at all-wavelengths are shown to allow for highly coherent continua that can be nonlinearly shaped using appropriate initial conditions. In contrast, supercontinua generated in the anomalous dispersion regime are shown to exhibit large fluctuations in the temporal and spectral domains that can be controlled using a carefully chosen seed. A particular example of this is the first experimental observation of the Peregrine soliton which constitutes a prototype of optical rogue-waves.

Appropriate wage rate and estimate of labour availability for a public works based safety net in Cambodia

Employment generating public works (EGPW) are an important part of Royal Government of Cambodia’s (RGC’s) strategy being developed through Council for Agriculture and Rural Development (CARD) to develop a comprehensive social safety net (SSN) to provide a measure of protection from shocks for the poor and vulnerable and to contribute to poverty alleviation through short-term unskilled employment.

Nonlinear spectral shaping and optical rogue events in fiber-based systems

We provide an overview of our recent work on the shaping and stability of optical continua in the long pulse regime. Fibers with normal group-velocity dispersion at all-wavelengths are shown to allow for highly coherent continua that can be nonlinearly shaped using appropriate initial conditions. In contrast, supercontinua generated in the anomalous dispersion regime are shown to exhibit large fluctuations in the temporal and spectral domains that can be controlled using a carefully chosen seed. A particular example of this is the first experimental observation of the Peregrine soliton which constitutes a prototype of optical rogue-waves. © 2012 Elsevier Inc. All rights reserved.

Subject identification via ECG fiducial-based systems:influence of the type of QT interval correction

Electrocardiography (ECG) has been recently proposed as biometric trait for identification purposes. Intra-individual variations of ECG might affect identification performance. These variations are mainly due to Heart Rate Variability (HRV). In particular, HRV causes changes in the QT intervals along the ECG waveforms. This work is aimed at analysing the influence of seven QT interval correction methods (based on population models) on the performance of ECG-fiducial-based identification systems. In addition, we have also considered the influence of training set size, classifier, classifier ensemble as well as the number of consecutive heartbeats in a majority voting scheme. The ECG signals used in this study were collected from thirty-nine subjects within the Physionet open access database. Public domain software was used for fiducial points detection. Results suggested that QT correction is indeed required to improve the performance. However, there is no clear choice among the seven explored approaches for QT correction (identification rate between 0.97 and 0.99). MultiLayer Perceptron and Support Vector Machine seemed to have better generalization capabilities, in terms of classification performance, with respect to Decision Tree-based classifiers. No such strong influence of the training-set size and the number of consecutive heartbeats has been observed on the majority voting scheme.