A Research of Resource Scheduling Algorithm Based on Hadoop

With the development of electronic devices, more and more mobile clients are connected to the Internet and they generate massive data every day. We live in an age of “Big Data”, and every day we generate hundreds of million magnitude data. By analyzing the data and making prediction, we can carry out better development plan. Unfortunately, traditional computation framework cannot meet the demand, so the Hadoop would be put forward. First the paper introduces the background and development status of Hadoop, compares the MapReduce in Hadoop 1.0 and YARN in Hadoop 2.0, and analyzes the advantages and disadvantages of them. Because the resource management module is the core role of YARN, so next the paper would research about the resource allocation module including the resource management, resource allocation algorithm, resource preemption model and the whole resource scheduling process from applying resource to finishing allocation. Also it would introduce the FIFO Scheduler, Capacity Scheduler, and Fair Scheduler and compare them. The main work has been done in this paper is researching and analyzing the Dominant Resource Fair algorithm of YARN, putting forward a maximum resource utilization algorithm based on Dominant Resource Fair algorithm. The paper also provides a suggestion to improve the unreasonable facts in resource preemption model. Emphasizing “fairness” during resource allocation is the core concept of Dominant Resource Fair algorithm of YARM. Because the cluster is multiple users and multiple resources, so the user’s resource request is multiple too. The DRF algorithm would divide the user’s resources into dominant resource and normal resource. For a user, the dominant resource is the one whose share is highest among all the request resources, others are normal resource. The DRF algorithm requires the dominant resource share of each user being equal. But for these cases where different users’ dominant resource amount differs greatly, emphasizing “fairness” is not suitable and can’t promote the resource utilization of the cluster. By analyzing these cases, this thesis puts forward a new allocation algorithm based on DRF. The new algorithm takes the “fairness” into consideration but not the main principle. Maximizing the resource utilization is the main principle and goal of the new algorithm. According to comparing the result of the DRF and new algorithm based on DRF, we found that the new algorithm has more high resource utilization than DRF. The last part of the thesis is to install the environment of YARN and use the Scheduler Load Simulator (SLS) to simulate the cluster environment.

Macroalgal Defenses Against Herbivory: Causes and Consequences of Intraspecific Variation

In marine benthic communities, herbivores consume a considerable proportion of primary producer biomass and, thus, generate selection for the evolution of resistance traits. According to the theory of plant defenses, resistance traits are costly to produce and, consequently, inducible resistance traits are adaptive in conditions of variable herbivory, while in conditions of constant/strong herbivory constitutive resistance traits are selected for. The evolution of resistance plasticity may be constrained by the costs of resistance or lack of genetic variation in resistance. Furthermore, resource allocation to induced resistance may be affected by higher trophic levels preying on herbivores. I studied the resistance to herbivory of a foundation species, the brown alga Fucus vesiculosus. By using factorial field experiments, I explored the effects of herbivores and fish predators on growth and resistance of the alga in two seasons. I explored genetic variation in and allocation costs of resistance traits as well as their chemical basis and their effects on herbivore performance. Using a field experiment I tested if induced resistance spreads via water-borne cues from one individual to another in relevant ecological conditions. I found that in the northern Baltic Sea F. vesiculosus communities, strength of three trophic interactions strongly vary among seasons. The highly synchronized summer reproduction of herbivores promoted their escape from the top-down control of fish predators in autumn. This resulted into large grazing losses in algal stands. In spring, herbivore densities were low and regulated by fish, which, thus,enhanced algal growth. The resistance of algae to herbivory increased with an increase in constitutive phlorotannin content. Furthermore, individuals adopted induced resistance when grazed and when exposed to water-borne cues originating from grazing of conspecific algae both in the laboratory and in field conditions. Induced resistance was adopted to a lesser extent in the presence of fish predators. The results in this thesis indicate that inducible resistance in F. vesiculosus is an adaptation to varying herbivory in the northern Baltic Sea. The costs of resistance and strong seasonality of herbivory have likely contributed to the evolution of this defense strategy. My findings also show that fish predators have positive cascading effects on F. vesiculosus which arise via reduced herbivory but possibly also through reduced resource allocation to resistance. I further found evidence that the spread of resistance via water-borne cues also occurs in ecologically realistic conditions in natural marine sublittoral. Thus, water-borne induction may enable macroalgae to cope with the strong grazing pressure characteristic of marine benthic communities. The results presented here show that seasonality can have pronounced effects on the biotic interactions in marine benthic communities and thereafter influence the evolution of resistance traits in primary producers.