Optimizing the reliability and resource efficiency of MapReduce-based systems

Debido al gran incremento de datos digitales que ha tenido lugar en los últimos años, ha surgido un nuevo paradigma de computación paralela para el procesamiento eficiente de grandes volúmenes de datos. Muchos de los sistemas basados en este paradigma, también llamados sistemas de computación intensiva de datos, siguen el modelo de programación de Google MapReduce. La principal ventaja de los sistemas MapReduce es que se basan en la idea de enviar la computación donde residen los datos, tratando de proporcionar escalabilidad y eficiencia. En escenarios libres de fallo, estos sistemas generalmente logran buenos resultados. Sin embargo, la mayoría de escenarios donde se utilizan, se caracterizan por la existencia de fallos. Por tanto, estas plataformas suelen incorporar características de tolerancia a fallos y fiabilidad. Por otro lado, es reconocido que las mejoras en confiabilidad vienen asociadas a costes adicionales en recursos. Esto es razonable y los proveedores que ofrecen este tipo de infraestructuras son conscientes de ello. No obstante, no todos los enfoques proporcionan la misma solución de compromiso entre las capacidades de tolerancia a fallo (o de manera general, las capacidades de fiabilidad) y su coste. Esta tesis ha tratado la problemática de la coexistencia entre fiabilidad y eficiencia de los recursos en los sistemas basados en el paradigma MapReduce, a través de metodologías que introducen el mínimo coste, garantizando un nivel adecuado de fiabilidad. Para lograr esto, se ha propuesto: (i) la formalización de una abstracción de detección de fallos; (ii) una solución alternativa a los puntos únicos de fallo de estas plataformas, y, finalmente, (iii) un nuevo sistema de asignación de recursos basado en retroalimentación a nivel de contenedores. Estas contribuciones genéricas han sido evaluadas tomando como referencia la arquitectura Hadoop YARN, que, hoy en día, es la plataforma de referencia en la comunidad de los sistemas de computación intensiva de datos. En la tesis se demuestra cómo todas las contribuciones de la misma superan a Hadoop YARN tanto en fiabilidad como en eficiencia de los recursos utilizados. ABSTRACT Due to the increase of huge data volumes, a new parallel computing paradigm to process big data in an efficient way has arisen. Many of these systems, called dataintensive computing systems, follow the Google MapReduce programming model. The main advantage of these systems is based on the idea of sending the computation where the data resides, trying to provide scalability and efficiency. In failure-free scenarios, these frameworks usually achieve good results. However, these ones are not realistic scenarios. Consequently, these frameworks exhibit some fault tolerance and dependability techniques as built-in features. On the other hand, dependability improvements are known to imply additional resource costs. This is reasonable and providers offering these infrastructures are aware of this. Nevertheless, not all the approaches provide the same tradeoff between fault tolerant capabilities (or more generally, reliability capabilities) and cost. In this thesis, we have addressed the coexistence between reliability and resource efficiency in MapReduce-based systems, looking for methodologies that introduce the minimal cost and guarantee an appropriate level of reliability. In order to achieve this, we have proposed: (i) a formalization of a failure detector abstraction; (ii) an alternative solution to single points of failure of these frameworks, and finally (iii) a novel feedback-based resource allocation system at the container level. Finally, our generic contributions have been instantiated for the Hadoop YARN architecture, which is the state-of-the-art framework in the data-intensive computing systems community nowadays. The thesis demonstrates how all our approaches outperform Hadoop YARN in terms of reliability and resource efficiency.

Multiple Robots, Single Operator: Considerations About Information And Commanding

Multiple robot, single operator scenarios suppose a challenge in terms of human factors. Two relevant issues are keeping the situational awareness and managing the workload of operators. In order to address these problems, this work analyses the management of information and commands in multi-robot missions. About the information, this paper proposes a selection based on mission and operator states. Regarding the commands, this work reflects about the levels of automation and the methods of commanding.

Antioxidative Defense System, Pigment Composition, and Photosynthetic Efficiency in Two Wheat Cultivars Subjected to Drought1

We analyzed antioxidative defenses, photosynthesis, and pigments (especially xanthophyll-cycle components) in two wheat (Triticum durum Desf.) cultivars, Adamello and Ofanto, during dehydration and rehydration to determine the difference in their sensitivities to drought and to elucidate the role of different protective mechanisms against oxidative stress. Drought caused a more pronounced inhibition in growth and photosynthetic rates in the more sensitive cv Adamello compared with the relatively tolerant cv Ofanto. During dehydration the glutathione content decreased in both wheat cultivars, but only cv Adamello showed a significant increase in glutathione reductase and hydrogen peroxide-glutathione peroxidase activities. The activation states of two sulfhydryl-containing chloroplast enzymes, NADP+-dependent glyceraldehyde-3-phosphate dehydrogenase and fructose-1,6-bisphosphatase, were maintained at control levels during dehydration and rehydration in both cultivars. This indicates that the defense systems involved are efficient in the protection of sulfhydryl groups against oxidation. Drought did not cause significant effects on lipid peroxidation. Upon dehydration, a decline in chlorophyll a, lutein, neoxanthin, and β-carotene contents, and an increase in the pool of de-epoxidized xanthophyll-cycle components (i.e. zeaxanthin and antheraxanthin), were evident only in cv Adamello. Accordingly, after exposure to drought, cv Adamello showed a larger reduction in the actual photosystem II photochemical efficiency and a higher increase in nonradiative energy dissipation than cv Ofanto. Although differences in zeaxanthin content were not sufficient to explain the difference in drought tolerance between the two cultivars, zeaxanthin formation may be relevant in avoiding irreversible damage to photosystem II in the more sensitive cultivar.

The impact of financial development and market conditions on investment-cash flow sensitivity: global evidence

In this study, we analyze the impact of financial development and market conditions on investment-cash flow sensitivity during the 2006-2014 for 76 countries. First, the results show a relationship between investment-cash flow sensitivity and an index of financial development and its components. Second, 68 countries are affected by the 2008-2009 financial crisis, but only 16 countries exhibit a higher investment-cash flow sensitivity during the crisis. Third, investment-cash flow sensitivity is lower in countries with a larger primary debt market, while the size of the primary equity market has no impact. Finally, analyzing investment-cash flow sensitivity over time, we find lower sensitivity during years associated with higher primary debt market activity.

New Approaches for Teaching Soil and Rock Mechanics Using Information and Communication Technologies

Soil and rock mechanics are disciplines with a strong conceptual and methodological basis. Initially, when engineering students study these subjects, they have to understand new theoretical phenomena, which are explained through mathematical and/or physical laws (e.g. consolidation process, water flow through a porous media). In addition to the study of these phenomena, students have to learn how to carry out estimations of soil and rock parameters in laboratories according to standard tests. Nowadays, information and communication technologies (ICTs) provide a unique opportunity to improve the learning process of students studying the aforementioned subjects. In this paper, we describe our experience of the incorporation of ICTs into the classical teaching-learning process of soil and rock mechanics and explain in detail how we have successfully developed various initiatives which, in summary, are: (a) implementation of an online social networking and microblogging service (using Twitter) for gradually sending key concepts to students throughout the semester (gradual learning); (b) detailed online virtual laboratory tests for a delocalized development of lab practices (self-learning); (c) integration of different complementary learning resources (e.g. videos, free software, technical regulations, etc.) using an open webpage. The complementary use to the classical teaching-learning process of these ICT resources has been highly satisfactory for students, who have positively evaluated this new approach.

Saving the Monopsony: Exclusivity, Innovation and Market Power in the Media Sector. Research Papers in Law, 7/2006

From the Introduction. The Media Sector has experienced a technological revolution in the last 15 years. Digital encoding of television signals made possible a more efficient use of the radiospectrum. Digital terrestrial television (hereinafter, “DTT”) allows now for the reception of a significant number of free-to-air channels.1 Moreover, the use of new transmission platforms (hereinafter,“platforms”), namely cable and direct-to-home satellite (hereinafter, “DTH”) paved the way for the arrival in Europe of pay-TV operators, which finance their activities mainly via subscription fees. This changing technological landscape is subject to further evolution in the near future, as incumbent telecommunications operators become increasingly interested in making available broadcasting content2 as part of their broadband offer and 3G mobile handsets can be used for the reception of TV signals....The present paper seeks to ascertain whether the Commission “regulatory approach” towards the exclusive sale of premium content is a sound one, in particular in view of the constant technological evolution outlined above. The assumptions underlying landmark Commission decisions will be compared with recent developments of the media sector in Italy. In the NewsCorp./Telepiù case, decided in 2003, the Commission imposed very strict conditions to allow the merger giving birth to Sky Italia, on the assumption that the operation created a lasting near-monopsony in the different upstream markets for the acquisition of premium intervened against the media conglomerate Mediaset (which controls, inter alia, the main three private free-to-air channels in Italy) for an alleged abuse of dominant position.17 In fact, and contrary to the forecasts made by the Commission, Mediaset was in a position to acquire the broadcasting rights of the main Italian football teams, thereby excluding the incumbent (and near-monopolist) pay-TV operator, Sky Italia. This may go to show that the reality of the sector is more complex and evolves faster than one may infer from the Commission practice, thus putting into question its stance regarding exclusivity. The experience of the evolution of the Italian media sector will be used as the starting point for the evaluation of alternative regulatory options.