Benchmarking of distributed computing engines spark and GraphLab for big data analytics

In this paper we evaluate and compare two representativeand popular distributed processing engines for large scalebig data analytics, Spark and graph based engine GraphLab. Wedesign a benchmark suite including representative algorithmsand datasets to compare the performances of the computingengines, from performance aspects of running time, memory andCPU usage, network and I/O overhead. The benchmark suite istested on both local computer cluster and virtual machines oncloud. By varying the number of computers and memory weexamine the scalability of the computing engines with increasingcomputing resources (such as CPU and memory). We also runcross-evaluation of generic and graph based analytic algorithmsover graph processing and generic platforms to identify thepotential performance degradation if only one processing engineis available. It is observed that both computing engines showgood scalability with increase of computing resources. WhileGraphLab largely outperforms Spark for graph algorithms, ithas close running time performance as Spark for non-graphalgorithms. Additionally the running time with Spark for graphalgorithms over cloud virtual machines is observed to increaseby almost 100% compared to over local computer clusters.

A compact polymer optical fibre ultrasound detector

Polymer optical fibre (POF) is a relatively new and novel technology that presents an innovative approach for ultrasonic endoscopic applications. Currently, piezo electric transducers are the typical detectors of choice, albeit possessing a limited bandwidth due to their resonant nature and a sensitivity that decreases proportionally to their size. Optical fibres provide immunity from electromagnetic interference and POF in particular boasts more suitable physical characteristics than silica optical fibre. The most important of these are lower acoustic impedance, a reduced Young's Modulus and a higher acoustic sensitivity than single-mode silica fibre at both 1 MHz and 10 MHz. POF therefore offers an interesting alternative to existing technology. Intrinsic fibre structures such as Bragg gratings and Fabry-Perot cavities may be inscribed into the fibre core using UV lasers. These gratings are a modulation of the refractive index of the fibre core and provide the advantages of high reflectivity, customisable bandwidth and point detection. We present a compact in fibre ultrasonic point detector based upon a POF Bragg grating (POFBG) sensor. We demonstrate that the detector is capable of leaving a laboratory environment by using connectorised fibre sensors and make a case for endoscopic ultrasonic detection through use of a mounting structure that better mimics the environment of an endoscopic probe. We measure the effects of water immersion upon POFBGs and analyse the ultrasonic response for 1, 5 and 10 MHz.

On the stability of conventional and nano-structured carbon-based catalysts in the oxidative dehydrogenation of ethylbenzene under industrially relevant conditions

Relevant carbon-based materials, home-made carbon-silica hybrids, commercial activated carbon, and nanostructured multi-walled carbon nanotubes (MWCNT) were tested in the oxidative dehydrogenation of ethylbenzene (EB). Special attention was given to the reaction conditions, using a relatively concentrated EB feed (10 vol.% EB), and limited excess of O2 (O 2:EB = 0.6) in order to work at full oxygen conversion and consequently avoid O2 in the downstream processing and recycle streams. The temperature was varied between 425 and 475 °C, that is about 150-200 °C lower than that of the commercial steam dehydrogenation process. The stability was evaluated from runs of 60 h time on stream. Under the applied reactions conditions, all the carbon-based materials are apparently stable in the first 15 h time on stream. The effect of the gasification/burning was significantly visible only after this period where most of them fully decomposes. The carbon of the hybrids decomposes completely rendering the silica matrix and the activated carbon bed is fully consumed. Nano structured MWCNT is the most stable; the structure resists the demanding reaction conditions showing an EB conversion of ∼30% (but deactivating) with a steady selectivity of ∼80%. The catalyst stability under the ODH reaction conditions is predicted from the combustion apparent activation energies. © 2014 Elsevier Ltd. All rights reserved.

Scattering of the laser writing beam in photonic crystal fibre

Point-by-point fibre grating fabrication by femtosecond laser pulses requires tight focusing of the pulses into the core of the fibre. This condition is not easily satisfied in photonic crystal fibres (PCFs) due to the pulse scattering by the holes. In this letter, we present a numerical model of propagation of tightly focused laser beam through PCF in a typical experimental setup. We investigate impact of the numerical aperture of the beam and hole refractive index on the beam scattering and identify optimal conditions for relating the findings to the requirements of grating fabrication. The results explain and quantify recent experimental grating inscription techniques and are indicative of birefringence observed in long-period gratings written by femtosecond laser pulses. © 2010 Elsevier Ltd. All rights reserved.

Raman fibre laser based amplification in coherent transmission systems

The thesis presents a detailed study of different Raman fibre laser (RFL) based amplification techniques and their applications in long-haul/unrepeatered coherent transmission systems. RFL based amplifications techniques were characterised from different aspects, including signal/noise power distributions, relative intensity noise (RIN), mode structures of induced Raman fibre lasers, and so on. It was found for the first time that RFL based amplification techniques could be divided into three categories in terms of the fibre laser regime, which were Fabry-Perot fibre laser with two FBGs, weak Fabry-Perot fibre laser with one FBG and very low reflection near the input, and random distributed feedback (DFB) fibre laser with one FBG. It was also found that lowering the reflection near the input could mitigate the RIN of the signal significantly, thanks to the reduced efficiency of the Stokes shift from the FW-propagated pump. In order to evaluate the transmission performance, different RFL based amplifiers were evaluated and optimised in long-haul coherent transmission systems. The results showed that Fabry-Perot fibre laser based amplifier with two FBGs gave >4.15 dB Q factor penalty using symmetrical bidirectional pumping, as the RIN of the signal was increased significantly. However, random distributed feedback fibre laser based amplifier with one FBG could mitigate the RIN of the signal, which enabled the use of bidirectional second order pumping and consequently give the best transmission performance up to 7915 km. Furthermore, using random DFB fibre laser based amplifier was proved to be effective to combat the nonlinear impairment, and the maximum reach was enhanced by >28% in mid-link single/dual band optical phase conjugator (OPC) transmission systems. In addition, unrepeatered transmission over >350 km fibre length using RFL based amplification technique were presented experimentally using DP-QPSK and DP-16QAM transmitter.

Integration and querying of heterogeneous, autonomous, distributed database systems

Today, databases have become an integral part of information systems. In the past two decades, we have seen different database systems being developed independently and used in different applications domains. Today's interconnected networks and advanced applications, such as data warehousing, data mining & knowledge discovery and intelligent data access to information on the Web, have created a need for integrated access to such heterogeneous, autonomous, distributed database systems. Heterogeneous/multidatabase research has focused on this issue resulting in many different approaches. However, a single, generally accepted methodology in academia or industry has not emerged providing ubiquitous intelligent data access from heterogeneous, autonomous, distributed information sources. ^ This thesis describes a heterogeneous database system being developed at High-performance Database Research Center (HPDRC). A major impediment to ubiquitous deployment of multidatabase technology is the difficulty in resolving semantic heterogeneity. That is, identifying related information sources for integration and querying purposes. Our approach considers the semantics of the meta-data constructs in resolving this issue. The major contributions of the thesis work include: (i) providing a scalable, easy-to-implement architecture for developing a heterogeneous multidatabase system, utilizing Semantic Binary Object-oriented Data Model (Sem-ODM) and Semantic SQL query language to capture the semantics of the data sources being integrated and to provide an easy-to-use query facility; (ii) a methodology for semantic heterogeneity resolution by investigating into the extents of the meta-data constructs of component schemas. This methodology is shown to be correct, complete and unambiguous; (iii) a semi-automated technique for identifying semantic relations, which is the basis of semantic knowledge for integration and querying, using shared ontologies for context-mediation; (iv) resolutions for schematic conflicts and a language for defining global views from a set of component Sem-ODM schemas; (v) design of a knowledge base for storing and manipulating meta-data and knowledge acquired during the integration process. This knowledge base acts as the interface between integration and query processing modules; (vi) techniques for Semantic SQL query processing and optimization based on semantic knowledge in a heterogeneous database environment; and (vii) a framework for intelligent computing and communication on the Internet applying the concepts of our work. ^

Data partitioning and replication management in distributed geographic information system (GIS) database

This research presents several components encompassing the scope of the objective of Data Partitioning and Replication Management in Distributed GIS Database. Modern Geographic Information Systems (GIS) databases are often large and complicated. Therefore data partitioning and replication management problems need to be addresses in development of an efficient and scalable solution. ^ Part of the research is to study the patterns of geographical raster data processing and to propose the algorithms to improve availability of such data. These algorithms and approaches are targeting granularity of geographic data objects as well as data partitioning in geographic databases to achieve high data availability and Quality of Service(QoS) considering distributed data delivery and processing. To achieve this goal a dynamic, real-time approach for mosaicking digital images of different temporal and spatial characteristics into tiles is proposed. This dynamic approach reuses digital images upon demand and generates mosaicked tiles only for the required region according to user's requirements such as resolution, temporal range, and target bands to reduce redundancy in storage and to utilize available computing and storage resources more efficiently. ^ Another part of the research pursued methods for efficient acquiring of GIS data from external heterogeneous databases and Web services as well as end-user GIS data delivery enhancements, automation and 3D virtual reality presentation. ^ There are vast numbers of computing, network, and storage resources idling or not fully utilized available on the Internet. Proposed "Crawling Distributed Operating System "(CDOS) approach employs such resources and creates benefits for the hosts that lend their CPU, network, and storage resources to be used in GIS database context. ^ The results of this dissertation demonstrate effective ways to develop a highly scalable GIS database. The approach developed in this dissertation has resulted in creation of TerraFly GIS database that is used by US government, researchers, and general public to facilitate Web access to remotely-sensed imagery and GIS vector information. ^

Statistical analysis and meta-analysis of microarray data

The microarray technology provides a high-throughput technique to study gene expression. Microarrays can help us diagnose different types of cancers, understand biological processes, assess host responses to drugs and pathogens, find markers for specific diseases, and much more. Microarray experiments generate large amounts of data. Thus, effective data processing and analysis are critical for making reliable inferences from the data. ^ The first part of dissertation addresses the problem of finding an optimal set of genes (biomarkers) to classify a set of samples as diseased or normal. Three statistical gene selection methods (GS, GS-NR, and GS-PCA) were developed to identify a set of genes that best differentiate between samples. A comparative study on different classification tools was performed and the best combinations of gene selection and classifiers for multi-class cancer classification were identified. For most of the benchmarking cancer data sets, the gene selection method proposed in this dissertation, GS, outperformed other gene selection methods. The classifiers based on Random Forests, neural network ensembles, and K-nearest neighbor (KNN) showed consistently god performance. A striking commonality among these classifiers is that they all use a committee-based approach, suggesting that ensemble classification methods are superior. ^ The same biological problem may be studied at different research labs and/or performed using different lab protocols or samples. In such situations, it is important to combine results from these efforts. The second part of the dissertation addresses the problem of pooling the results from different independent experiments to obtain improved results. Four statistical pooling techniques (Fisher inverse chi-square method, Logit method. Stouffer's Z transform method, and Liptak-Stouffer weighted Z-method) were investigated in this dissertation. These pooling techniques were applied to the problem of identifying cell cycle-regulated genes in two different yeast species. As a result, improved sets of cell cycle-regulated genes were identified. The last part of dissertation explores the effectiveness of wavelet data transforms for the task of clustering. Discrete wavelet transforms, with an appropriate choice of wavelet bases, were shown to be effective in producing clusters that were biologically more meaningful. ^

Design and implementation of a multisite data repository for pediatric epilepsy

This dissertation established a software-hardware integrated design for a multisite data repository in pediatric epilepsy. A total of 16 institutions formed a consortium for this web-based application. This innovative fully operational web application allows users to upload and retrieve information through a unique human-computer graphical interface that is remotely accessible to all users of the consortium. A solution based on a Linux platform with My-SQL and Personal Home Page scripts (PHP) has been selected. Research was conducted to evaluate mechanisms to electronically transfer diverse datasets from different hospitals and collect the clinical data in concert with their related functional magnetic resonance imaging (fMRI). What was unique in the approach considered is that all pertinent clinical information about patients is synthesized with input from clinical experts into 4 different forms, which were: Clinical, fMRI scoring, Image information, and Neuropsychological data entry forms. A first contribution of this dissertation was in proposing an integrated processing platform that was site and scanner independent in order to uniformly process the varied fMRI datasets and to generate comparative brain activation patterns. The data collection from the consortium complied with the IRB requirements and provides all the safeguards for security and confidentiality requirements. An 1-MR1-based software library was used to perform data processing and statistical analysis to obtain the brain activation maps. Lateralization Index (LI) of healthy control (HC) subjects in contrast to localization-related epilepsy (LRE) subjects were evaluated. Over 110 activation maps were generated, and their respective LIs were computed yielding the following groups: (a) strong right lateralization: (HC=0%, LRE=18%), (b) right lateralization: (HC=2%, LRE=10%), (c) bilateral: (HC=20%, LRE=15%), (d) left lateralization: (HC=42%, LRE=26%), e) strong left lateralization: (HC=36%, LRE=31%). Moreover, nonlinear-multidimensional decision functions were used to seek an optimal separation between typical and atypical brain activations on the basis of the demographics as well as the extent and intensity of these brain activations. The intent was not to seek the highest output measures given the inherent overlap of the data, but rather to assess which of the many dimensions were critical in the overall assessment of typical and atypical language activations with the freedom to select any number of dimensions and impose any degree of complexity in the nonlinearity of the decision space.

In vivo characterization of myocardial tissue post myocardial infarction using combined fluorescence and diffuse reflectance spectroscopy

Accurately assessing the extent of myocardial tissue injury induced by Myocardial infarction (MI) is critical to the planning and optimization of MI patient management. With this in mind, this study investigated the feasibility of using combined fluorescence and diffuse reflectance spectroscopy to characterize a myocardial infarct at the different stages of its development. An animal study was conducted using twenty male Sprague-Dawley rats with MI. In vivo fluorescence spectra at 337 nm excitation and diffuse reflectance between 400 nm and 900 nm were measured from the heart using a portable fiber-optic spectroscopic system. Spectral acquisition was performed on (1) the normal heart region; (2) the region immediately surrounding the infarct; and (3) the infarcted region—one, two, three and four weeks into MI development. The spectral data were divided into six subgroups according to the histopathological features associated with various degrees/severities of myocardial tissue injury as well as various stages of myocardial tissue remodeling, post infarction. Various data processing and analysis techniques were employed to recognize the representative spectral features corresponding to various histopathological features associated with myocardial infarction. The identified spectral features were utilized in discriminant analysis to further evaluate their effectiveness in classifying tissue injuries induced by MI. In this study, it was observed that MI induced significant alterations (p < 0.05) in the diffuse reflectance spectra, especially between 450 nm and 600 nm, from myocardial tissue within the infarcted and surrounding regions. In addition, MI induced a significant elevation in fluorescence intensities at 400 and 460 nm from the myocardial tissue from the same regions. The extent of these spectral alterations was related to the duration of the infarction. Using the spectral features identified, an effective tissue injury classification algorithm was developed which produced a satisfactory overall classification result (87.8%). The findings of this research support the concept that optical spectroscopy represents a useful tool to non-invasively determine the in vivo pathophysiological features of a myocardial infarct and its surrounding tissue, thereby providing valuable real-time feedback to surgeons during various surgical interventions for MI.

Makespan minimization in a job shop with a batch processing machine

A job shop with one batch processing and several discrete machines is analyzed. Given a set of jobs, their process routes, processing requirements, and size, the objective is to schedule the jobs such that the makespan is minimized. The batch processing machine can process a batch of jobs as long as the machine capacity is not violated. The batch processing time is equal to the longest processing job in the batch. The problem under study can be represented as Jm:batch:Cmax. If no batches were formed, the scheduling problem under study reduces to the classical job shop scheduling problem (i.e. Jm:: Cmax), which is known to be NP-hard. This research extends the scheduling literature by combining Jm::Cmax with batch processing. The primary contributions are the mathematical formulation, a new network representation and several solution approaches. The problem under study is observed widely in metal working and other industries, but received limited or no attention due to its complexity. A novel network representation of the problem using disjunctive and conjunctive arcs, and a mathematical formulation are proposed to minimize the makespan. Besides that, several algorithms, like batch forming heuristics, dispatching rules, Modified Shifting Bottleneck, Tabu Search (TS) and Simulated Annealing (SA), were developed and implemented. An experimental study was conducted to evaluate the proposed heuristics, and the results were compared to those from a commercial solver (i.e., CPLEX). TS and SA, with the combination of MWKR-FF as the initial solution, gave the best solutions among all the heuristics proposed. Their results were close to CPLEX; and for some larger instances, with total operations greater than 225, they were competitive in terms of solution quality and runtime. For some larger problem instances, CPLEX was unable to report a feasible solution even after running for several hours. Between SA and the experimental study indicated that SA produced a better average Cmax for all instances. The solution approaches proposed will benefit practitioners to schedule a job shop (with both discrete and batch processing machines) more efficiently. The proposed solution approaches are easier to implement and requires short run times to solve large problem instances.

Stream dissolved organic matter bioavailability and composition in watersheds underlain with discontinuous permafrost

We examined the impact of permafrost on dissolved organic matter (DOM) composition in Caribou-Poker Creeks Research Watershed (CPCRW), a watershed underlain with discontinuous permafrost, in interior Alaska. We analyzed long term data from watersheds underlain with varying degrees of permafrost, sampled springs and thermokarsts, used fluorescence spectroscopy, and measured the bioavailabity of dissolved organic carbon (DOC). Permafrost driven patterns in hydrology and vegetation influenced DOM patterns in streams, with the stream draining the high permafrost watershed having higher DOC and dissolved organic nitrogen (DON) concentrations, higher DOC:- DON and greater specific ultraviolet absorbance (SUVA) than the streams draining the low and medium permafrost watersheds. Streams, springs and thermokarsts exhibited a wide range of DOC and DON concentrations (1.5–37.5 mgC/L and 0.14–1.26 mgN/L, respectively), DOC:DON (7.1–42.8) and SUVA (1.5–4.7 L mgC-1 m-1). All sites had a high proportion of humic components, a low proportion of protein components, and a low fluorescence index value (1.3–1.4), generally consistent with terrestrially derivedDOM. Principal component analysis revealed distinct groups in our fluorescence data determined by diagenetic processing and DOM source. The proportion of bioavailable DOC ranged from 2 to 35%, with the proportion of tyrosine- and tryptophan-like fluorophores in the DOM being a major predictor of DOC loss (p\0.05, R2 = 0.99). Our results indicate that the degradation of permafrost in CPCRW will result in a decrease in DOC and DON concentrations, a decline in DOC:DON, and a reduction in SUVA, possibly accompanied by