Alleviating ‘overfitting’ via genetically-regularised neural network

A hybrid genetic algorithm/scaled conjugate gradient regularisation method is designed to alleviate ANN `over-fitting'. In application to day-ahead load forecasting, the proposed algorithm performs better than early-stopping and Bayesian regularisation, showing promising initial results.

Dynamic Covariance Models for Multivariate Financial Time Series

The accurate prediction of time-changing covariances is an important problem in the modeling of multivariate financial data. However, some of the most popular models suffer from a) overfitting problems and multiple local optima, b) failure to capture shifts in market conditions and c) large computational costs. To address these problems we introduce a novel dynamic model for time-changing covariances. Over-fitting and local optima are avoided by following a Bayesian approach instead of computing point estimates. Changes in market conditions are captured by assuming a diffusion process in parameter values, and finally computationally efficient and scalable inference is performed using particle filters. Experiments with financial data show excellent performance of the proposed method with respect to current standard models.

Gaussian process volatility model

The prediction of time-changing variances is an important task in the modeling of financial data. Standard econometric models are often limited as they assume rigid functional relationships for the evolution of the variance. Moreover, functional parameters are usually learned by maximum likelihood, which can lead to over-fitting. To address these problems we introduce GP-Vol, a novel non-parametric model for time-changing variances based on Gaussian Processes. This new model can capture highly flexible functional relationships for the variances. Furthermore, we introduce a new online algorithm for fast inference in GP-Vol. This method is much faster than current offline inference procedures and it avoids overfitting problems by following a fully Bayesian approach. Experiments with financial data show that GP-Vol performs significantly better than current standard alternatives.

Simultaneous Learning of Non-Linear Manifold and Dynamical Models for High-Dimensional Time Series

The goal of this work is to learn a parsimonious and informative representation for high-dimensional time series. Conceptually, this comprises two distinct yet tightly coupled tasks: learning a low-dimensional manifold and modeling the dynamical process. These two tasks have a complementary relationship as the temporal constraints provide valuable neighborhood information for dimensionality reduction and conversely, the low-dimensional space allows dynamics to be learnt efficiently. Solving these two tasks simultaneously allows important information to be exchanged mutually. If nonlinear models are required to capture the rich complexity of time series, then the learning problem becomes harder as the nonlinearities in both tasks are coupled. The proposed solution approximates the nonlinear manifold and dynamics using piecewise linear models. The interactions among the linear models are captured in a graphical model. The model structure setup and parameter learning are done using a variational Bayesian approach, which enables automatic Bayesian model structure selection, hence solving the problem of over-fitting. By exploiting the model structure, efficient inference and learning algorithms are obtained without oversimplifying the model of the underlying dynamical process. Evaluation of the proposed framework with competing approaches is conducted in three sets of experiments: dimensionality reduction and reconstruction using synthetic time series, video synthesis using a dynamic texture database, and human motion synthesis, classification and tracking on a benchmark data set. In all experiments, the proposed approach provides superior performance.

Cross-Domain Multitask Learning with Latent Probit Models

Learning multiple tasks across heterogeneous domains is a challenging problem since the feature space may not be the same for different tasks. We assume the data in multiple tasks are generated from a latent common domain via sparse domain transforms and propose a latent probit model (LPM) to jointly learn the domain transforms, and the shared probit classifier in the common domain. To learn meaningful task relatedness and avoid over-fitting in classification, we introduce sparsity in the domain transforms matrices, as well as in the common classifier. We derive theoretical bounds for the estimation error of the classifier in terms of the sparsity of domain transforms. An expectation-maximization algorithm is derived for learning the LPM. The effectiveness of the approach is demonstrated on several real datasets.

A Largest Matching Area Approach to Image Denoising

Given the success of patch-based approaches to image denoising,this paper addresses the ill-posed problem of patch size selection.Large patch sizes improve noise robustness in the presence of good matches, but can also lead to artefacts in textured regions due to the rare patch effect; smaller patch sizes reconstruct details more accurately but risk over-fitting to the noise in uniform regions. We propose to jointly optimize each matching patch’s identity and size for gray scale image denoising, and present several implementations.The new approach effectively selects the largest matching areas, subject to the constraints of the available data and noise level, to improve noise robustness. Experiments on standard test images demonstrate our approach’s ability to improve on fixed-size reconstruction, particularly at high noise levels, on smoother image regions.

Development and validation of a risk model for prediction of hazardous alcohol consumption in general practice attendees : the PredictAL study

Background: Little is known about the risk of progression to hazardous alcohol use in people currently drinking at safe limits. We aimed to develop a prediction model (predictAL) for the development of hazardous drinking in safe drinkers. Methods: A prospective cohort study of adult general practice attendees in six European countries and Chile followed up over 6 months. We recruited 10,045 attendees between April 2003 to February 2005. 6193 European and 2462 Chilean attendees recorded AUDIT scores below 8 in men and 5 in women at recruitment and were used in modelling risk. 38 risk factors were measured to construct a risk model for the development of hazardous drinking using stepwise logistic regression. The model was corrected for over fitting and tested in an external population. The main outcome was hazardous drinking defined by an AUDIT score >= 8 in men and >= 5 in women. Results: 69.0% of attendees were recruited, of whom 89.5% participated again after six months. The risk factors in the final predictAL model were sex, age, country, baseline AUDIT score, panic syndrome and lifetime alcohol problem. The predictAL model's average c-index across all six European countries was 0.839 (95% CI 0.805, 0.873). The Hedge's g effect size for the difference in log odds of predicted probability between safe drinkers in Europe who subsequently developed hazardous alcohol use and those who did not was 1.38 (95% CI 1.25, 1.51). External validation of the algorithm in Chilean safe drinkers resulted in a c-index of 0.781 (95% CI 0.717, 0.846) and Hedge's g of 0.68 (95% CI 0.57, 0.78). Conclusions: The predictAL risk model for development of hazardous consumption in safe drinkers compares favourably with risk algorithms for disorders in other medical settings and can be a useful first step in prevention of alcohol misuse.

Génération et reconnaissance de rythmes au moyen de réseaux de neurones à réservoir

Les fichiers sons qui accompagne mon document sont au format midi. Le programme que nous avons développés pour ce travail est en language Python.

Développement d’un modèle centré sur l’individu des déplacements du caribou, du loup et de l’orignal, et de leurs interactions, en forêt boréale aménagée

Le caribou forestier est une espèce menacée au Canada, la principale hypothèse au déclin des populations étant l’intensification de la prédation provoquée par les perturbations anthropiques du paysage. Afin de faire face à cette situation, il est nécessaire d’étudier et comprendre l’impact de l’environnement sur les interactions prédateur-proies entre le caribou et le loup, ainsi qu’avec l’orignal, qui est sa principale proie alternative. Pour cela, cette thèse présente la conception d’un modèle centré sur l’individu des déplacements de ces trois espèces en fonction de leur environnement, dont résulteront les interactions prédateur-proies. Afin de permettre l’application de ce modèle sur de longues périodes, et donc pour un environnement changeant, une méthodologie a été développée, qui s’articule atour de deux aspects principaux. Tout d’abord, la notion de niveaux d’émergence est introduite, permettant d’ordonner les comportements observables du système selon leurs interdépendances, afin de choisir comme trait du modèle un com- portement correspondant au domaine d’applicabilité visé. Ordonner les comportements selon leurs niveaux d’émergence permet également d’identifier la redondance entre les patrons, qui peut être à l’origine d’un phénomène de sur-apprentissage lorsqu’ils sont utilisés lors de la calibration. Dans un second temps, un nouveau protocole pour la calibration et la validation du ou des traits choisis à l’aide des niveaux d’émergence, nommé réplication de système basé sur l’individu (Individual Based System Replication - IBSRtion) est également présenté. Ce protocole met l’emphase sur la modélisation directe, contrairement au principal protocole existant, la modélisation orientée patrons (Pattern Oriented Modelling - POM), et permet une approche empirique en générant artificiellement des données non disponibles ou ne pouvant être récoltées par des études de terrains. IBSRtion a également l’avantage de pouvoir être intégrée dans POM, afin de contribuer à la création d’une méthodologie universelle pour la conception de modèles centrés sur l’individu. Le processus de conception de ce modèle aura entre autre permis de faire une synthèse des connaissances et d’identifier certaines lacunes. Une étude visant à palier le manque de connaissances satisfaisantes sur les réponses comportementales à court-terme des proies face au risque de prédation a notamment permis d’observer que celles-ci sont une combinaison de comportements chroniques et éphémères, et que les mécanismes qui en sont à l’origine sont complexes et non-linéaires. Le résultat de ce travail est un modèle complexe utilisant de nombreux sous-modèles, et calibré de façon empirique, applicable à une grande variété d’environnements. Ce modèle a permis de tester l’impact de l’enfeuillement sur les relations prédateur-proies. Des simulations ont été effectuées pour différentes quantités d’enfeuillement, suivant deux configurations spatiales différentes. Les résultats de simulation suggèrent que des plans d’aménagement considérant également l’habitat de l’orignal pourraient être bénéfiques pour le caribou forestier, car ils permettraient d’améliorer la ségrégation spatiale entre les deux espèces, et donc entre le caribou et le loup. En le couplant avec un module de naissances et de morts naturelles ainsi qu’un modèle d’évolution du paysage, ce modèle permettra par la suite d’évaluer l’impact de plans d’aménagement forestier sur la viabilité des populations de caribou forestier.

Does increased hydrochemical model complexity decrease robustness?

The aim of this study was, within a sensitivity analysis framework, to determine if additional model complexity gives a better capability to model the hydrology and nitrogen dynamics of a small Mediterranean forested catchment or if the additional parameters cause over-fitting. Three nitrogen-models of varying hydrological complexity were considered. For each model, general sensitivity analysis (GSA) and Generalized Likelihood Uncertainty Estimation (GLUE) were applied, each based on 100,000 Monte Carlo simulations. The results highlighted the most complex structure as the most appropriate, providing the best representation of the non-linear patterns observed in the flow and streamwater nitrate concentrations between 1999 and 2002. Its 5% and 95% GLUE bounds, obtained considering a multi-objective approach, provide the narrowest band for streamwater nitrogen, which suggests increased model robustness, though all models exhibit periods of inconsistent good and poor fits between simulated outcomes and observed data. The results confirm the importance of the riparian zone in controlling the short-term (daily) streamwater nitrogen dynamics in this catchment but not the overall flux of nitrogen from the catchment. It was also shown that as the complexity of a hydrological model increases over-parameterisation occurs, but the converse is true for a water quality model where additional process representation leads to additional acceptable model simulations. Water quality data help constrain the hydrological representation in process-based models. Increased complexity was justifiable for modelling river-system hydrochemistry. Increased complexity was justifiable for modelling river-system hydrochemistry.

Modelling methodology and forecast failure

We analyse by simulation the impact of model-selection strategies (sometimes called pre-testing) on forecast performance in both constant-and non-constant-parameter processes. Restricted, unrestricted and selected models are compared when either of the first two might generate the data. We find little evidence that strategies such as general-to-specific induce significant over-fitting, or thereby cause forecast-failure rejection rates to greatly exceed nominal sizes. Parameter non-constancies put a premium on correct specification, but in general, model-selection effects appear to be relatively small, and progressive research is able to detect the mis-specifications.

Prediction of wool knitwear pilling propensity using support vector machines

The propensity of wool knitwear to form entangled fiber balls, known as pills, on the surface is affected by a large number of factors. This study examines, for the first time, the application of the support vector machine (SVM) data mining tool to the pilling propensity prediction of wool knitwear. The results indicate that by using the binary classification method and the radial basis function (RBF) kernel function, the SVM is able to give high pilling propensity prediction accuracy for wool knitwear without data over-fitting. The study also found that the number of records available for each pill rating greatly affects the learning and prediction capability of SVM models.

Mixed transfer function neural networks for knowledge acquistition

Modeling helps to understand and predict the outcome of complex systems. Inductive modeling methodologies are beneficial for modeling the systems where the uncertainties involved in the system do not permit to obtain an accurate physical model. However inductive models, like artificial neural networks (ANNs), may suffer from a few drawbacks involving over-fitting and the difficulty to easily understand the model itself. This can result in user reluctance to accept the model or even complete rejection of the modeling results. Thus, it becomes highly desirable to make such inductive models more comprehensible and to automatically determine the model complexity to avoid over-fitting. In this paper, we propose a novel type of ANN, a mixed transfer function artificial neural network (MTFANN), which aims to improve the complexity fitting and comprehensibility of the most popular type of ANN (MLP - a Multilayer Perceptron).

Prediction of sepsis in the burn intensive care unit patient

Sepsis is a significant cause for multiple organ failure and death in the burn patient, yet identification in this population is confounded by chronic hypermetabolism and impaired immune function. The purpose of this study was twofold: 1) determine the ability of the systemic inflammatory response syndrome (SIRS) and American Burn Association (ABA) criteria to predict sepsis in the burn patient; and 2) develop a model representing the best combination of clinical predictors associated with sepsis in the same population. A retrospective, case-controlled, within-patient comparison of burn patients admitted to a single intensive care unit (ICU) was conducted for the period January 2005 to September 2010. Blood culture results were paired with clinical condition: "positive-sick"; "negative-sick", and "screening-not sick". Data were collected for the 72 hours prior to each blood culture. The most significant predictors were evaluated using logistic regression, Generalized Estimating Equations (GEE) and ROC area under the curve (AUC) analyses to assess model predictive ability. Bootstrapping methods were employed to evaluate potential model over-fitting. Fifty-nine subjects were included, representing 177 culture periods. SIRS criteria were not found to be associated with culture type, with an average of 98% of subjects meeting criteria in the 3 days prior. ABA sepsis criteria were significantly different among culture type only on the day prior (p = 0.004). The variables identified for the model included: heart rate>130 beats/min, mean blood pressure<60 mmHg, base deficit<-6 mEq/L, temperature>36°C, use of vasoactive medications, and glucose>150 mg/d1. The model was significant in predicting "positive culture-sick" and sepsis state, with AUC of 0.775 (p < 0.001) and 0.714 (p < .001), respectively; comparatively, the ABA criteria AUC was 0.619 (p = 0.028) and 0.597 (p = .035), respectively. SIRS criteria are not appropriate for identifying sepsis in the burn population. The ABA criteria perform better, but only for the day prior to positive blood culture results. The time period useful to diagnose sepsis using clinical criteria may be limited to 24 hours. A combination of predictors is superior to individual variable trends, yet algorithms or computer support will be necessary for the clinician to find such models useful. ^

Extreme learning machine for predicting HLA-peptide binding

Machine learning techniques have been recognized as powerful tools for learning from data. One of the most popular learning techniques, the Back-Propagation (BP) Artificial Neural Networks, can be used as a computer model to predict peptides binding to the Human Leukocyte Antigens (HLA). The major advantage of computational screening is that it reduces the number of wet-lab experiments that need to be performed, significantly reducing the cost and time. A recently developed method, Extreme Learning Machine (ELM), which has superior properties over BP has been investigated to accomplish such tasks. In our work, we found that the ELM is as good as, if not better than, the BP in term of time complexity, accuracy deviations across experiments, and most importantly - prevention from over-fitting for prediction of peptide binding to HLA.