3 resultados para HDFS bottleneck

em DigitalCommons@University of Nebraska - Lincoln


Relevância:

10.00% 10.00%

Publicador:

Resumo:

Right whales carry large populations of three ‘whale lice’ (Cyamus ovalis, Cyamus gracilis, Cyamus erraticus) that have no other hosts. We used sequence variation in the mitochondrial COI gene to ask (i) whether cyamid population structures might reveal associations among right whale individuals and subpopulations, (ii) whether the divergences of the three nominally conspecific cyamid species on North Atlantic, North Pacific, and southern right whales (Eubalaena glacialis, Eubalaena japonica, Eubalaena australis) might indicate their times of separation, and (iii) whether the shapes of cyamid gene trees might contain information about changes in the population sizes of right whales. We found high levels of nucleotide diversity but almost no population structure within oceans, indicating large effective population sizes and high rates of transfer between whales and subpopulations. North Atlantic and Southern Ocean populations of all three species are reciprocally monophyletic, and North Pacific C. erraticus is well separated from North Atlantic and southern C. erraticus. Mitochondrial clock calibrations suggest that these divergences occurred around 6 million years ago (Ma), and that the Eubalaena mitochondrial clock is very slow. North Pacific C. ovalis forms a clade inside the southern C. ovalis gene tree, implying that at least one right whale has crossed the equator in the Pacific Ocean within the last 1–2 million years (Myr). Low-frequency polymorphisms are more common than expected under neutrality for populations of constant size, but there is no obvious signal of rapid, interspecifically congruent expansion of the kind that would be expected if North Atlantic or southern right whales had experienced a prolonged population bottleneck within the last 0.5 Myr.

Relevância:

10.00% 10.00%

Publicador:

Resumo:

Hundreds of Terabytes of CMS (Compact Muon Solenoid) data are being accumulated for storage day by day at the University of Nebraska-Lincoln, which is one of the eight US CMS Tier-2 sites. Managing this data includes retaining useful CMS data sets and clearing storage space for newly arriving data by deleting less useful data sets. This is an important task that is currently being done manually and it requires a large amount of time. The overall objective of this study was to develop a methodology to help identify the data sets to be deleted when there is a requirement for storage space. CMS data is stored using HDFS (Hadoop Distributed File System). HDFS logs give information regarding file access operations. Hadoop MapReduce was used to feed information in these logs to Support Vector Machines (SVMs), a machine learning algorithm applicable to classification and regression which is used in this Thesis to develop a classifier. Time elapsed in data set classification by this method is dependent on the size of the input HDFS log file since the algorithmic complexities of Hadoop MapReduce algorithms here are O(n). The SVM methodology produces a list of data sets for deletion along with their respective sizes. This methodology was also compared with a heuristic called Retention Cost which was calculated using size of the data set and the time since its last access to help decide how useful a data set is. Accuracies of both were compared by calculating the percentage of data sets predicted for deletion which were accessed at a later instance of time. Our methodology using SVMs proved to be more accurate than using the Retention Cost heuristic. This methodology could be used to solve similar problems involving other large data sets.

Relevância:

10.00% 10.00%

Publicador:

Resumo:

Software product line (SPL) engineering offers several advantages in the development of families of software products such as reduced costs, high quality and a short time to market. A software product line is a set of software intensive systems, each of which shares a common core set of functionalities, but also differs from the other products through customization tailored to fit the needs of individual groups of customers. The differences between products within the family are well-understood and organized into a feature model that represents the variability of the SPL. Products can then be built by generating and composing features described in the feature model. Testing of software product lines has become a bottleneck in the SPL development lifecycle, since many of the techniques used in their testing have been borrowed from traditional software testing and do not directly take advantage of the similarities between products. This limits the overall gains that can be achieved in SPL engineering. Recent work proposed by both industry and the research community for improving SPL testing has begun to consider this problem, but there is still a need for better testing techniques that are tailored to SPL development. In this thesis, I make two primary contributions to software product line testing. First I propose a new definition for testability of SPLs that is based on the ability to re-use test cases between products without a loss of fault detection effectiveness. I build on this idea to identify elements of the feature model that contribute positively and/or negatively towards SPL testability. Second, I provide a graph based testing approach called the FIG Basis Path method that selects products and features for testing based on a feature dependency graph. This method should increase our ability to re-use results of test cases across successive products in the family and reduce testing effort. I report the results of a case study involving several non-trivial SPLs and show that for these objects, the FIG Basis Path method is as effective as testing all products, but requires us to test no more than 24% of the products in the SPL.