Managing Large Data Sets Using Support Vector Machines

Hundreds of Terabytes of CMS (Compact Muon Solenoid) data are being accumulated for storage day by day at the University of Nebraska-Lincoln, which is one of the eight US CMS Tier-2 sites. Managing this data includes retaining useful CMS data sets and clearing storage space for newly arriving data by deleting less useful data sets. This is an important task that is currently being done manually and it requires a large amount of time. The overall objective of this study was to develop a methodology to help identify the data sets to be deleted when there is a requirement for storage space. CMS data is stored using HDFS (Hadoop Distributed File System). HDFS logs give information regarding file access operations. Hadoop MapReduce was used to feed information in these logs to Support Vector Machines (SVMs), a machine learning algorithm applicable to classification and regression which is used in this Thesis to develop a classifier. Time elapsed in data set classification by this method is dependent on the size of the input HDFS log file since the algorithmic complexities of Hadoop MapReduce algorithms here are O(n). The SVM methodology produces a list of data sets for deletion along with their respective sizes. This methodology was also compared with a heuristic called Retention Cost which was calculated using size of the data set and the time since its last access to help decide how useful a data set is. Accuracies of both were compared by calculating the percentage of data sets predicted for deletion which were accessed at a later instance of time. Our methodology using SVMs proved to be more accurate than using the Retention Cost heuristic. This methodology could be used to solve similar problems involving other large data sets.

ENHANCEMENT OF THE FAA’s ON-LINE WILDLIFE AIRCRAFT STRIKE DATABASE WITH AN INTERACTIVE GRAPHICS CAPABILITY

Embry-Riddle Aeronautical University (Prescott, AZ, USA) was awarded a grant from the William J. Hughes FAA Technical Center in October 1999 to develop and maintain a web site dealing with a wide variety of airport safety wildlife concerns. Initially, the web site enabled users to access related topics such as wildlife management (at/near airports), bird identification information, FAA wildlife management guidelines, education, pictures, current news, upcoming meetings and training, available jobs and discussion/forum sections. In April 2001, the web site was augmented with an on-line wildlife strike report (FAA Form 5200-7). Upon submittal on-line, “quick look” email notifications are sent to concerned government personnel. The distribution of these emails varies as to whether there was damage, human injuries/fatalities, and whether feather remains were collected and will be sent to the Smithsonian Institution for identification. In July 2002, a real-time on-line query system was incorporated to allow federal and local government agencies, airport and operator personnel, and USDA and airport wildlife biologists to access this database (which as of June 2005 contains 68,288 researched strike reports added to at a rate of approximately 500 strike reports/month) to formulate strategies to reduce the hazards wildlife present to aviation. To date (June 2005), over 15,000 on-line real-time queries were processed. In June 2004, ERAU was authorized to develop a graphical interface to this on-line query system. Current capabilities include mapping strikes (by species) on the US map, each of the contiguous 48 state maps (with AK and HI being added), and airport diagrams of the major metropolitan airports as well as the next 46 airports with the most reported strikes The latter capability depicts strikes by runway in plan as well as in elevation view. Currently under development is the ability to view time-sequenced strikes on the US map. This extensive graphical interface will give analysts the ability to view strike patterns with a wide variety of variables including species, seasons, migration patterns, etc. on US and state maps and airport diagrams.