Managing Large Data Sets Using Support Vector Machines

Hundreds of Terabytes of CMS (Compact Muon Solenoid) data are being accumulated for storage day by day at the University of Nebraska-Lincoln, which is one of the eight US CMS Tier-2 sites. Managing this data includes retaining useful CMS data sets and clearing storage space for newly arriving data by deleting less useful data sets. This is an important task that is currently being done manually and it requires a large amount of time. The overall objective of this study was to develop a methodology to help identify the data sets to be deleted when there is a requirement for storage space. CMS data is stored using HDFS (Hadoop Distributed File System). HDFS logs give information regarding file access operations. Hadoop MapReduce was used to feed information in these logs to Support Vector Machines (SVMs), a machine learning algorithm applicable to classification and regression which is used in this Thesis to develop a classifier. Time elapsed in data set classification by this method is dependent on the size of the input HDFS log file since the algorithmic complexities of Hadoop MapReduce algorithms here are O(n). The SVM methodology produces a list of data sets for deletion along with their respective sizes. This methodology was also compared with a heuristic called Retention Cost which was calculated using size of the data set and the time since its last access to help decide how useful a data set is. Accuracies of both were compared by calculating the percentage of data sets predicted for deletion which were accessed at a later instance of time. Our methodology using SVMs proved to be more accurate than using the Retention Cost heuristic. This methodology could be used to solve similar problems involving other large data sets.

NATIONWIDE ESTIMATES OF BLACKBIRDS AND STARLINGS

An estimated 538 million blackbirds and Starlings are found in the United States, based on the national cooperative blackbird/Starling winter roost survey conducted by the U.S. Fish and Wildlife Service during the 1974-75 winter period of December 20-February 15. Ap- proximately 74% or 398 million of these blackbirds and Starlings occurred in the Eastern States, including the tier from Minnesota to Louisiana; 26% or 139 million birds were in the West. The national roosting population in 1974-75 was composed of 11 species (Table 1) in the following approximate proportions: 38% Red-winged Blackbirds; 22% Common Grackles; 20% Starlings; 18% Brown-headed Cowbirds; 2% Brewer’s Blackbirds; and less than 1% six species combined (Rusty Blackbirds, Boat-tailed Grackles, Great-tailed Grackles, Tri-colored Black- birds, Yellow-headed Blackbirds, and Bronzed Cowbirds). (Some 2 million robins also were reported in the 1974-75 survey, though not solicited and therefore not tabulated, from 20 of the blackbird roosts in the Southeast.) The 1974-75 species proportions are similar to those found in the last nationwide winter survey (1969-70). In the 1963-64 national winter survey, Redwings made up 33% and Common Grackles 31% of the total population.