Designing Regularizers and Architectures for Recurrent Neural Networks

Cette thèse contribue a la recherche vers l'intelligence artificielle en utilisant des méthodes connexionnistes. Les réseaux de neurones récurrents sont un ensemble de modèles séquentiels de plus en plus populaires capable en principe d'apprendre des algorithmes arbitraires. Ces modèles effectuent un apprentissage en profondeur, un type d'apprentissage machine. Sa généralité et son succès empirique en font un sujet intéressant pour la recherche et un outil prometteur pour la création de l'intelligence artificielle plus générale. Le premier chapitre de cette thèse donne un bref aperçu des sujets de fonds: l'intelligence artificielle, l'apprentissage machine, l'apprentissage en profondeur et les réseaux de neurones récurrents. Les trois chapitres suivants couvrent ces sujets de manière de plus en plus spécifiques. Enfin, nous présentons quelques contributions apportées aux réseaux de neurones récurrents. Le chapitre \ref{arxiv1} présente nos travaux de régularisation des réseaux de neurones récurrents. La régularisation vise à améliorer la capacité de généralisation du modèle, et joue un role clé dans la performance de plusieurs applications des réseaux de neurones récurrents, en particulier en reconnaissance vocale. Notre approche donne l'état de l'art sur TIMIT, un benchmark standard pour cette tâche. Le chapitre \ref{cpgp} présente une seconde ligne de travail, toujours en cours, qui explore une nouvelle architecture pour les réseaux de neurones récurrents. Les réseaux de neurones récurrents maintiennent un état caché qui représente leurs observations antérieures. L'idée de ce travail est de coder certaines dynamiques abstraites dans l'état caché, donnant au réseau une manière naturelle d'encoder des tendances cohérentes de l'état de son environnement. Notre travail est fondé sur un modèle existant; nous décrivons ce travail et nos contributions avec notamment une expérience préliminaire.

Deriving matrix of peptide-MHC interactions in diabetic mouse by genetic algorithm

Finding motifs that can elucidate rules that govern peptide binding to medically important receptors is important for screening targets for drugs and vaccines. This paper focuses on elucidation of peptide binding to I-A(g7) molecule of the non-obese diabetic (NOD) mouse - an animal model for insulin-dependent diabetes mellitus (IDDM). A number of proposed motifs that describe peptide binding to I-A(g7) have been proposed. These motifs results from independent experimental studies carried out on small data sets. Testing with multiple data sets showed that each of the motifs at best describes only a subset of the solution space, and these motifs therefore lack generalization ability. This study focuses on seeking a motif with higher generalization ability so that it can predict binders in all A(g7) data sets with high accuracy. A binding score matrix representing peptide binding motif to A(g7) was derived using genetic algorithm (GA). The evolved score matrix significantly outperformed previously reported

The evolution of learning systems:to Bayes or not to be

Bayesian algorithms pose a limit to the performance learning algorithms can achieve. Natural selection should guide the evolution of information processing systems towards those limits. What can we learn from this evolution and what properties do the intermediate stages have? While this question is too general to permit any answer, progress can be made by restricting the class of information processing systems under study. We present analytical and numerical results for the evolution of on-line algorithms for learning from examples for neural network classifiers, which might include or not a hidden layer. The analytical results are obtained by solving a variational problem to determine the learning algorithm that leads to maximum generalization ability. Simulations using evolutionary programming, for programs that implement learning algorithms, confirm and expand the results. The principal result is not just that the evolution is towards a Bayesian limit. Indeed it is essentially reached. In addition we find that evolution is driven by the discovery of useful structures or combinations of variables and operators. In different runs the temporal order of the discovery of such combinations is unique. The main result is that combinations that signal the surprise brought by an example arise always before combinations that serve to gauge the performance of the learning algorithm. This latter structures can be used to implement annealing schedules. The temporal ordering can be understood analytically as well by doing the functional optimization in restricted functional spaces. We also show that there is data suggesting that the appearance of these traits also follows the same temporal ordering in biological systems. © 2006 American Institute of Physics.

A machine learning approach for recognising woody plants on railway trackbeds

The purpose of this work in progress study was to test the concept of recognising plants using images acquired by image sensors in a controlled noise-free environment. The presence of vegetation on railway trackbeds and embankments presents potential problems. Woody plants (e.g. Scots pine, Norway spruce and birch) often establish themselves on railway trackbeds. This may cause problems because legal herbicides are not effective in controlling them; this is particularly the case for conifers. Thus, if maintenance administrators knew the spatial position of plants along the railway system, it may be feasible to mechanically harvest them. Primary data were collected outdoors comprising around 700 leaves and conifer seedlings from 11 species. These were then photographed in a laboratory environment. In order to classify the species in the acquired image set, a machine learning approach known as Bag-of-Features (BoF) was chosen. Irrespective of the chosen type of feature extraction and classifier, the ability to classify a previously unseen plant correctly was greater than 85%. The maintenance planning of vegetation control could be improved if plants were recognised and localised. It may be feasible to mechanically harvest them (in particular, woody plants). In addition, listed endangered species growing on the trackbeds can be avoided. Both cases are likely to reduce the amount of herbicides, which often is in the interest of public opinion. Bearing in mind that natural objects like plants are often more heterogeneous within their own class rather than outside it, the results do indeed present a stable classification performance, which is a sound prerequisite in order to later take the next step to include a natural background. Where relevant, species can also be listed under the Endangered Species Act.

A kernel-based two-class classifier for imbalanced data sets

Many kernel classifier construction algorithms adopt classification accuracy as performance metrics in model evaluation. Moreover, equal weighting is often applied to each data sample in parameter estimation. These modeling practices often become problematic if the data sets are imbalanced. We present a kernel classifier construction algorithm using orthogonal forward selection (OFS) in order to optimize the model generalization for imbalanced two-class data sets. This kernel classifier identification algorithm is based on a new regularized orthogonal weighted least squares (ROWLS) estimator and the model selection criterion of maximal leave-one-out area under curve (LOO-AUC) of the receiver operating characteristics (ROCs). It is shown that, owing to the orthogonalization procedure, the LOO-AUC can be calculated via an analytic formula based on the new regularized orthogonal weighted least squares parameter estimator, without actually splitting the estimation data set. The proposed algorithm can achieve minimal computational expense via a set of forward recursive updating formula in searching model terms with maximal incremental LOO-AUC value. Numerical examples are used to demonstrate the efficacy of the algorithm.

A fast linear-in-the-parameters classifier construction algorithm using orthogonal forward selection to minimize leave-one-out misclassification rate

We propose a simple and computationally efficient construction algorithm for two class linear-in-the-parameters classifiers. In order to optimize model generalization, a forward orthogonal selection (OFS) procedure is used for minimizing the leave-one-out (LOO) misclassification rate directly. An analytic formula and a set of forward recursive updating formula of the LOO misclassification rate are developed and applied in the proposed algorithm. Numerical examples are used to demonstrate that the proposed algorithm is an excellent alternative approach to construct sparse two class classifiers in terms of performance and computational efficiency.

An LDA and probability-based classifier for the diagnosis of Alzheimer's Disease from structural MRI

In this paper a custom classification algorithm based on linear discriminant analysis and probability-based weights is implemented and applied to the hippocampus measurements of structural magnetic resonance images from healthy subjects and Alzheimer’s Disease sufferers; and then attempts to diagnose them as accurately as possible. The classifier works by classifying each measurement of a hippocampal volume as healthy controlsized or Alzheimer’s Disease-sized, these new features are then weighted and used to classify the subject as a healthy control or suffering from Alzheimer’s Disease. The preliminary results obtained reach an accuracy of 85.8% and this is a similar accuracy to state-of-the-art methods such as a Naive Bayes classifier and a Support Vector Machine. An advantage of the method proposed in this paper over the aforementioned state of the art classifiers is the descriptive ability of the classifications it produces. The descriptive model can be of great help to aid a doctor in the diagnosis of Alzheimer’s Disease, or even further the understand of how Alzheimer’s Disease affects the hippocampus.

Learning generalization in problem solving by a blue-fronted parrot (Amazona aestiva)

Pepperberg (The Alex studies: cognitive and communicative abilities of gray parrots. Harvard University Press, Cambridge;1999) showed that some of the complex cognitive capabilities found in primates are also present in psittacine birds. Through the replication of an experiment performed with cotton-top tamarins (Saguinus oedipus oedipus) by Hauser et al. (Anim Behav 57:565-582; 1999), we examined a blue-fronted parrot`s (Amazona aestiva) ability to generalize the solution of a particular problem in new but similar cases. Our results show that, at least when it comes to solving this particular problem, our parrot subject exhibited learning generalization capabilities resembling the tamarins`.

An empirical evaluation of the Random Forests classifier models for variable selection in a large-scale lung cancer case-control study

Random Forests™ is reported to be one of the most accurate classification algorithms in complex data analysis. It shows excellent performance even when most predictors are noisy and the number of variables is much larger than the number of observations. In this thesis Random Forests was applied to a large-scale lung cancer case-control study. A novel way of automatically selecting prognostic factors was proposed. Also, synthetic positive control was used to validate Random Forests method. Throughout this study we showed that Random Forests can deal with large number of weak input variables without overfitting. It can account for non-additive interactions between these input variables. Random Forests can also be used for variable selection without being adversely affected by collinearities. ^ Random Forests can deal with the large-scale data sets without rigorous data preprocessing. It has robust variable importance ranking measure. Proposed is a novel variable selection method in context of Random Forests that uses the data noise level as the cut-off value to determine the subset of the important predictors. This new approach enhanced the ability of the Random Forests algorithm to automatically identify important predictors for complex data. The cut-off value can also be adjusted based on the results of the synthetic positive control experiments. ^ When the data set had high variables to observations ratio, Random Forests complemented the established logistic regression. This study suggested that Random Forests is recommended for such high dimensionality data. One can use Random Forests to select the important variables and then use logistic regression or Random Forests itself to estimate the effect size of the predictors and to classify new observations. ^ We also found that the mean decrease of accuracy is a more reliable variable ranking measurement than mean decrease of Gini. ^

A theoretical and experimental account of n-tuple classifier performance

The n-tuple recognition method is briefly reviewed, summarizing the main theoretical results. Large-scale experiments carried out on Stat-Log project datasets confirm this method as a viable competitor to more popular methods due to its speed, simplicity, and accuracy on the majority of a wide variety of classification problems. A further investigation into the failure of the method on certain datasets finds the problem to be largely due to a mismatch between the scales which describe generalization and data sparseness.

Differential impact of posterior lesions in the left and right hemisphere on visual category learning and generalization to contrast reversal

Hemispheric differences in the learning and generalization of pattern categories were explored in two experiments involving sixteen patients with unilateral posterior, cerebral lesions in the left (LH) or right (RH) hemisphere. In each experiment participants were first trained to criterion in a supervised learning paradigm to categorize a set of patterns that either consisted of simple geometric forms (Experiment 1) or unfamiliar grey-level images (Experiment 2). They were then tested for their ability to generalize acquired categorical knowledge to contrast-reversed versions of the learning patterns. The results showed that RH lesions impeded category learning of unfamiliar grey-level images more severely than LH lesions, whereas this relationship appeared reversed for categories defined by simple geometric forms. With regard to generalization to contrast reversal, categorization performance of LH and RH patients was unaffected in the case of simple geometric forms. However, generalization to of contrast-reversed grey-level images distinctly deteriorated for patients with LH lesions relative to those with RH lesions, with the latter (but not the former) being consistently unable to identify the pattern manipulation. These findings suggest a differential use of contrast information in the representation of pattern categories in the two hemispheres. Such specialization appears in line with previous distinctions between a predominantly lefthemispheric, abstract-analytical and a righthemispheric, specific-holistic representation of object categories, and their prediction of a mandatory representation of contrast polarity in the RH. Some implications for the well-established dissociation of visual disorders for the recognition of faces and letters are discussed.

Measuring centrality by a generalization of degree

Network analysis has emerged as a key technique in communication studies, economics, geography, history and sociology, among others. A fundamental issue is how to identify key nodes in a network, for which purpose a number of centrality measures have been developed. This paper proposes a new parametric family of centrality measures called generalized degree. It is based on the idea that a relationship to a more interconnected node contributes to centrality in a greater extent than a connection to a less central one. Generalized degree improves on degree by redistributing its sum over the network with the consideration of the global structure. Application of the measure is supported by a set of basic properties. A sufficient condition is given for generalized degree to be rank monotonic, excluding counter-intuitive changes in the centrality ranking after certain modifications of the network. The measure has a graph interpretation and can be calculated iteratively. Generalized degree is recommended to apply besides degree since it preserves most favorable attributes of degree, but better reflects the role of the nodes in the network and has an increased ability to distinguish between their importance.

The ability of the BANA test to detect different levels of P. gingivalis, T. denticola and T. forsythia

The aim of this study was to evaluate the ability of the BANA Test to detect different levels of Porphyromonas gingivalis, Treponema denticola and Tannerella forsythia or their combinations in subgingival samples at the initial diagnosis and after periodontal therapy. Periodontal sites with probing depths between 5-7 mm and clinical attachment level between 5-10 mm, from 53 subjects with chronic periodontitis, were sampled in four periods: initial diagnosis (T0), immediately (T1), 45 (T2) and 60 days (T3) after scaling and root planing. BANA Test and Checkerboard DNA-DNA hybridization identified red complex species in the subgingival biofilm. In all experimental periods, the highest frequencies of score 2 (Checkerboard DNA-DNA hybridization) for P. gingivalis, T. denticola and T. forsythia were observed when strong enzymatic activity (BANA) was present (p < 0.01). The best agreement was observed at initial diagnosis. The BANA Test sensitivity was 95.54% (T0), 65.18% (T1), 65.22% (T2) and 50.26% (T3). The specificity values were 12.24% (T0), 57.38% (T1), 46.27% (T2) and 53.48% (T3). The BANA Test is more effective for the detection of red complex pathogens when the bacterial levels are high, i.e. in the initial diagnosis of chronic periodontitis.

Working at night and work ability among nursing personnel: when precarious employment makes the diVerence

Purpose To test the association between night work and work ability, and verify whether the type of contractual employment has any inXuence over this association. Methods Permanent workers (N = 642) and workers with precarious jobs (temporary contract or outsourced; N = 552) were interviewed and Wlled out questionnaires concerning work hours and work ability index. They were classiWed into: never worked at night, ex-night workers, currently working up to Wve nights, and currently working at least six nights/2-week span. Results After adjusting for socio-demography and work variables, current night work was signiWcantly associated with inadequate WAI (vs. day work with no experience in night work) only for precarious workers (OR 2.00, CI 1.01- 3.95 and OR 1.85, CI 1.09-3.13 for those working up to Wve nights and those working at least six nights in 2 weeks, respectively). Conclusions Unequal opportunities at work and little experience in night work among precarious workers may explain their higher susceptibility to night work