Neural networks

An overview of neural networks, covering multilayer perceptrons, radial basis functions, constructive algorithms, Kohonen and K-means unupervised algorithms, RAMnets, first and second order training methods, and Bayesian regularisation methods.

Capacity of the upstart algorithm

The storage capacity of multilayer networks with overlapping receptive fields is investigated for a constructive algorithm within a one-step replica symmetry breaking (RSB) treatment. We find that the storage capacity increases logarithmically with the number of hidden units K without saturating the Mitchison-Durbin bound. The slope of the logarithmic increase decays exponentionally with the stability with which the patterns have been stored.

Classification of oximetry signals using Bayesian neural networks to assist in the detection of obstructive sleep apnoea syndrome

In the present study, multilayer perceptron (MLP) neural networks were applied to help in the diagnosis of obstructive sleep apnoea syndrome (OSAS). Oxygen saturation (SaO2) recordings from nocturnal pulse oximetry were used for this purpose. We performed time and spectral analysis of these signals to extract 14 features related to OSAS. The performance of two different MLP classifiers was compared: maximum likelihood (ML) and Bayesian (BY) MLP networks. A total of 187 subjects suspected of suffering from OSAS took part in the study. Their SaO2 signals were divided into a training set with 74 recordings and a test set with 113 recordings. BY-MLP networks achieved the best performance on the test set with 85.58% accuracy (87.76% sensitivity and 82.39% specificity). These results were substantially better than those provided by ML-MLP networks, which were affected by overfitting and achieved an accuracy of 76.81% (86.42% sensitivity and 62.83% specificity). Our results suggest that the Bayesian framework is preferred to implement our MLP classifiers. The proposed BY-MLP networks could be used for early OSAS detection. They could contribute to overcome the difficulties of nocturnal polysomnography (PSG) and thus reduce the demand for these studies.

Noise, regularizers, and unrealizable scenarios in online learning from restricted training sets

We study the dynamics of on-line learning in multilayer neural networks where training examples are sampled with repetition and where the number of examples scales with the number of network weights. The analysis is carried out using the dynamical replica method aimed at obtaining a closed set of coupled equations for a set of macroscopic variables from which both training and generalization errors can be calculated. We focus on scenarios whereby training examples are corrupted by additive Gaussian output noise and regularizers are introduced to improve the network performance. The dependence of the dynamics on the noise level, with and without regularizers, is examined, as well as that of the asymptotic values obtained for both training and generalization errors. We also demonstrate the ability of the method to approximate the learning dynamics in structurally unrealizable scenarios. The theoretical results show good agreement with those obtained by computer simulations.

The theory of on-line learning: a statistical physics approach

In this paper we review recent theoretical approaches for analysing the dynamics of on-line learning in multilayer neural networks using methods adopted from statistical physics. The analysis is based on monitoring a set of macroscopic variables from which the generalisation error can be calculated. A closed set of dynamical equations for the macroscopic variables is derived analytically and solved numerically. The theoretical framework is then employed for defining optimal learning parameters and for analysing the incorporation of second order information into the learning process using natural gradient descent and matrix-momentum based methods. We will also briefly explain an extension of the original framework for analysing the case where training examples are sampled with repetition.

DVMS 1.5: A user manual (the data visualization and modeling system)

The data available during the drug discovery process is vast in amount and diverse in nature. To gain useful information from such data, an effective visualisation tool is required. To provide better visualisation facilities to the domain experts (screening scientist, biologist, chemist, etc.),we developed a software which is based on recently developed principled visualisation algorithms such as Generative Topographic Mapping (GTM) and Hierarchical Generative Topographic Mapping (HGTM). The software also supports conventional visualisation techniques such as Principal Component Analysis, NeuroScale, PhiVis, and Locally Linear Embedding (LLE). The software also provides global and local regression facilities . It supports regression algorithms such as Multilayer Perceptron (MLP), Radial Basis Functions network (RBF), Generalised Linear Models (GLM), Mixture of Experts (MoE), and newly developed Guided Mixture of Experts (GME). This user manual gives an overview of the purpose of the software tool, highlights some of the issues to be taken care while creating a new model, and provides information about how to install & use the tool. The user manual does not require the readers to have familiarity with the algorithms it implements. Basic computing skills are enough to operate the software.

Techniques to improve forecasting models: applications to energy demand and price

This thesis is a study of three techniques to improve performance of some standard fore-casting models, application to the energy demand and prices. We focus on forecasting demand and price one-day ahead. First, the wavelet transform was used as a pre-processing procedure with two approaches: multicomponent-forecasts and direct-forecasts. We have empirically compared these approaches and found that the former consistently outperformed the latter. Second, adaptive models were introduced to continuously update model parameters in the testing period by combining ?lters with standard forecasting methods. Among these adaptive models, the adaptive LR-GARCH model was proposed for the fi?rst time in the thesis. Third, with regard to noise distributions of the dependent variables in the forecasting models, we used either Gaussian or Student-t distributions. This thesis proposed a novel algorithm to infer parameters of Student-t noise models. The method is an extension of earlier work for models that are linear in parameters to the non-linear multilayer perceptron. Therefore, the proposed method broadens the range of models that can use a Student-t noise distribution. Because these techniques cannot stand alone, they must be combined with prediction models to improve their performance. We combined these techniques with some standard forecasting models: multilayer perceptron, radial basis functions, linear regression, and linear regression with GARCH. These techniques and forecasting models were applied to two datasets from the UK energy markets: daily electricity demand (which is stationary) and gas forward prices (non-stationary). The results showed that these techniques provided good improvement to prediction performance.

Optical fibre sensors with applications in gas and biological sensing

This thesis describes the study of various grating based optical fibre sensors for applications in refractive index sensing. The sensitivity of these sensors has been studied and in some cases enhanced using novel techniques. The major areas of development are as follows. The sensitivity of long period gratings (LPGs) to surrounding medium refractive index (SRI) for various periods was investigated. The most sensitive period of LPG was found to be around 160 µm and this was due to the core mode coupling to a single cladding mode but phase matching at two wavelength locations, creating two attenuation peaks, close to the waveguide dispersion turning point. Large angle tilted fibre gratings (TFGs) have similar behaviour to LPGs, in that they couple to the co-propagating cladding modes. The tilted structure of the index modulation within the core of the fibre gives rise to a polarisation dependency, differing the large angle TFG from a LPG. Since the large angle TFG couple to the cladding mode they are SRI sensitive, the sensitivity to SRI can be further increased through cladding etching using HF acid. The thinning of the cladding layer caused a reordering of the cladding modes and shifted to more SRI sensitive cladding modes as the investigation discovered. In a SRI range of 1.36 to 1.40 a sensitivity of 506.9 nm/URI was achieved for the etched large angle TFG, which is greater than the dual resonance LPG. UV inscribed LPGs were coated with sol-gel materials with high RIs. The high RI of the coating caused an increase in cladding mode effective index which in turn caused an increase in the LPG sensitivity to SRI. LPGs of various periods of LPG were coated with sol-gel TiO2 and the optimal thickness was found to vary for each period. By coating of the already highly SRI sensitive 160µm period LPG (which is a dual resonance) with a sol-gel TiO2, the SRI sensitivity was further increased with a peak value of 1458 nm/URI, which was an almost 3 fold increase compared to the uncoated LPG. LPGs were also inscribed using a femtosecond laser which produced a highly focused index change which was no uniform throughout the core of the optical fibre. The inscription technique gave rise to a large polarisation sensitivity and the ability to couple to multiple azimuthal cladding mode sets, not seen with uniform UV inscribed gratings. Through coupling of the core mode to multiple sets of cladding modes, attenuation peaks with opposite wavelength shifts for increasing SRI was observed. Through combining this opposite wavelength shifts, a SRI sensitivity was achieved greater than any single observed attenuations peak. The maximum SRI achieved was 1680 nm/URI for a femtosecond inscribed LPG of period 400 µm. Three different types of surface plasmon resonance (SPR) sensors with a multilayer metal top coating were investigated in D shape optical fibre. The sensors could be separated into two types, utilized a pre UV inscribed tilted Bragg grating and the other employed a post UV exposure to generate surface relief grating structure. This surface perturbation aided the out coupling of light from the core but also changed the sensing mechanism from SPR to localised surface plasmon resonance (LSPR). This greatly increased the SRI sensitivity, compared to the SPR sensors; with the gold coated top layer surface relief sensor producing the largest SRI sensitivity of 2111.5nm/URI was achieved. While, the platinum and silver coated top layer surface relief sensors also gave high SRI sensitivities but also the ability to produce resonances in air (not previously seen with the SPR sensors). These properties were employed in two applications. The silver and platinum surface relief devices were used as gas sensors and were shown to be capable of detecting the minute RI change of different gases. The calculated maximum sensitivities produced were 1882.1dB/URI and 1493.5nm/URI for silver and platinum, respectively. Using a DFB laser and power meter a cheap alternative approach was investigated which showed the ability of the sensors to distinguish between different gases and flow rates of those gases. The gold surface relief sensor was coated in a with a bio compound called an aptamer and it was able to detect various concentrations of a biological compound called Thrombin, ranging from 1mM to as low as 10fM. A solution of 2M NaCl was found to give the best stripping results for Thrombin from the aptamer and showed the reusability of the sensor. The association and disassociation constants were calculated to be 1.0638×106Ms-1 and 0.2482s-1, respectively, showing the high affinity of the Aptamer to thrombin. This supports existing working stating that aptamers could be alternative to enzymes for chemical detection and also helps to explain the low detection limit of the gold surface relief sensor.

Disruption prediction at JET [Joint European Torus]

The sudden loss of the plasma magnetic confinement, known as disruption, is one of the major issue in a nuclear fusion machine as JET (Joint European Torus), Disruptions pose very serious problems to the safety of the machine. The energy stored in the plasma is released to the machine structure in few milliseconds resulting in forces that at JET reach several Mega Newtons. The problem is even more severe in the nuclear fusion power station where the forces are in the order of one hundred Mega Newtons. The events that occur during a disruption are still not well understood even if some mechanisms that can lead to a disruption have been identified and can be used to predict them. Unfortunately it is always a combination of these events that generates a disruption and therefore it is not possible to use simple algorithms to predict it. This thesis analyses the possibility of using neural network algorithms to predict plasma disruptions in real time. This involves the determination of plasma parameters every few milliseconds. A plasma boundary reconstruction algorithm, XLOC, has been developed in collaboration with Dr. D. Ollrien and Dr. J. Ellis capable of determining the plasma wall/distance every 2 milliseconds. The XLOC output has been used to develop a multilayer perceptron network to determine plasma parameters as ?i and q? with which a machine operational space has been experimentally defined. If the limits of this operational space are breached the disruption probability increases considerably. Another approach for prediction disruptions is to use neural network classification methods to define the JET operational space. Two methods have been studied. The first method uses a multilayer perceptron network with softmax activation function for the output layer. This method can be used for classifying the input patterns in various classes. In this case the plasma input patterns have been divided between disrupting and safe patterns, giving the possibility of assigning a disruption probability to every plasma input pattern. The second method determines the novelty of an input pattern by calculating the probability density distribution of successful plasma patterns that have been run at JET. The density distribution is represented as a mixture distribution, and its parameters arc determined using the Expectation-Maximisation method. If the dataset, used to determine the distribution parameters, covers sufficiently well the machine operational space. Then, the patterns flagged as novel can be regarded as patterns belonging to a disrupting plasma. Together with these methods, a network has been designed to predict the vertical forces, that a disruption can cause, in order to avoid that too dangerous plasma configurations are run. This network can be run before the pulse using the pre-programmed plasma configuration or on line becoming a tool that allows to stop dangerous plasma configuration. All these methods have been implemented in real time on a dual Pentium Pro based machine. The Disruption Prediction and Prevention System has shown that internal plasma parameters can be determined on-line with a good accuracy. Also the disruption detection algorithms showed promising results considering the fact that JET is an experimental machine where always new plasma configurations are tested trying to improve its performances.

Multiple conduction paths in boron δ-doped diamond structures

Impedance spectroscopy has been used to investigate conductivity within boron-doped diamond in an intrinsic/delta-doped/intrinsic (i-d-i) multilayer structure. For a 5 nm thick delta layer, three conduction pathways are observed, which can be assigned to transport within the delta layer and to two differing conduction paths in the i-layers adjoining the delta layer. For transport in the i-layers, thermal trapping/detrapping processes can be observed, and only at the highest temperature investigated (673 K) can transport due to a single conduction process be seen. Impedance spectroscopy is an ideal nondestructive tool for investigating the electrical characteristics of complex diamond structures.

Size-dependent radiation tolerance in ion irradiated TiN/AlN nanolayer films

Interface effects on ion-irradiation tolerance properties are investigated in nanolayered TiN/AlN films with individual layer thickness varied from 5 nm to 50 nm, prepared by pulsed laser deposition. Evolution of the microstructure and hardness of the multilayer films are examined on the specimens before and after He ion-implantation to a fluence of 4 × 10 m at 50 keV. The suppression of amorphization in AlN layers and the reduction of radiation-induced softening are observed in all nanolayer films. A clear size-dependent radiation tolerance characteristic is observed in the nanolayer films, i.e., the samples with the optimum layer thickness from 10 nm to 20 nm show the best ion irradiation tolerance properties, and a critical layer thickness of more than 5 nm is necessary to prevent severe intermixing. This study suggests that both the interface characteristics and the critical length scale (layer thickness) contribute to the reduction of the radiation-induced damages in nitride-based ceramic materials. © 2013 Elsevier B.V. All rights reserved.

Nanostructured Cu/Nb multilayers subjected to helium ion-irradiation

Helium ion-irradiation experiments have been performed in single layer Cu films, Nb films and Cu/Nb multilayer films with layer thickness varying from 2.5 nm to 100 nm each layer. Peak helium concentration approaches a few atomic percent with 6-9 displacement-per-atom in Cu and Nb. He bubbles were observed in single layer Cu and Nb films, as well as in Cu 100 nm/Nb 100 nm multilayers with helium bubbles aligned along layer interfaces. Helium bubbles are not resolved via transmission electron microscopy in Cu 2.5 nm/Nb 2.5 nm multilayers. These studies indicate that layer interface may play an important role in annihilating ion-irradiation induced defects such as vacancies and interstitials and have implications in improving the radiation tolerance of metallic materials using nanostructured multilayers. © 2007 Elsevier B.V. All rights reserved.

Strengthening mechanisms in nanostructured copper/304 stainless steel multilayers

Nanostructured Cu/304 stainless steel (SS) multilayers were prepared by magnetron sputtering. 304SS has a face-centered-cubic (fcc) structure in bulk. However, in the Cu/304SS multilayers, the 304SS layers exhibit the fcc structure for layer thickness of =5 nm in epitaxy with the neighboring fcc Cu. For 304SS layer thickness larger than 5 nm, body-centered-cubic (bcc) 304SS grains grow on top of the initial 5 nm fcc SS with the Kurdjumov-Sachs orientation relationship between bcc and fcc SS grains. The maximum hardness of Cu/304SS multilayers is about 5.5 GPa (factor of two enhancement compared to rule-of-mixtures hardness) at a layer thickness of 5 nm. Below 5 nm, hardness decreases with decreasing layer thickness. The peak hardness of fcc/fcc Cu/304SS multilayer is greater than that of Cu/Ni, even though the lattice-parameter mismatch between Cu and Ni is five times greater than that between Cu and 304SS. This result may primarily be attributed to the higher interface barrier stress for single-dislocation transmission across the {111} twinned interfaces in Cu/304SS as compared to the {100} interfaces in Cu/Ni.

Microwave properties of an inhomogeneous optically illuminated plasma in a microstrip gap

The optical illumination of a microstrip gap on a thick semiconductor substrate creates an inhomogeneous electron-hole plasma in the gap region. This allows the study of the propagation mechanism through the plasma region. This paper uses a multilayer plasma model to explain the origin of high losses in such structures. Measured results are shown up to 50 GHz and show good agreement with the simulated multilayer model. The model also allows the estimation of certain key parameters of the plasma, such as carrier density and diffusion length, which are difficult to measure by direct means. The detailed model validation performed here will enable the design of more complex microwave structures based on this architecture. While this paper focuses on monocrystalline silicon as the substrate, the model is easily adaptable to other semiconductor materials such as GaAs.

Subject identification via ECG fiducial-based systems:influence of the type of QT interval correction

Electrocardiography (ECG) has been recently proposed as biometric trait for identification purposes. Intra-individual variations of ECG might affect identification performance. These variations are mainly due to Heart Rate Variability (HRV). In particular, HRV causes changes in the QT intervals along the ECG waveforms. This work is aimed at analysing the influence of seven QT interval correction methods (based on population models) on the performance of ECG-fiducial-based identification systems. In addition, we have also considered the influence of training set size, classifier, classifier ensemble as well as the number of consecutive heartbeats in a majority voting scheme. The ECG signals used in this study were collected from thirty-nine subjects within the Physionet open access database. Public domain software was used for fiducial points detection. Results suggested that QT correction is indeed required to improve the performance. However, there is no clear choice among the seven explored approaches for QT correction (identification rate between 0.97 and 0.99). MultiLayer Perceptron and Support Vector Machine seemed to have better generalization capabilities, in terms of classification performance, with respect to Decision Tree-based classifiers. No such strong influence of the training-set size and the number of consecutive heartbeats has been observed on the majority voting scheme.