A generalized data windowing scheme for adaptive conjugate gradient algorithms

The performance of the modified adaptive conjugate gradient (CG) algorithms based on the iterative CG method for adaptive filtering is highly related to the ways of estimating the correlation matrix and the cross-correlation vector. The existing approaches of implementing the CG algorithms using the data windows of exponential form or sliding form result in either loss of convergence or increase in misadjustment. This paper presents and analyzes a new approach to the implementation of the CG algorithms for adaptive filtering by using a generalized data windowing scheme. For the new modified CG algorithms, we show that the convergence speed is accelerated, the misadjustment and tracking capability comparable to those of the recursive least squares (RLS) algorithm are achieved. Computer simulations demonstrated in the framework of linear system modeling problem show the improvements of the new modifications.

Improved particle swarm optimization-based artificial neural network for rainfall-runoff modeling

This paper presents the application of an improved particle swarm optimization (PSO) technique for training an artificial neural network (ANN) to predict water levels for the Heshui watershed, China. Daily values of rainfall and water levels from 1988 to 2000 were first analyzed using ANNs trained with the conjugate-gradient, gradient descent and Levenberg-Marquardt neural network (LM-NN) algorithms. The best results were obtained from LM-NN and these results were then compared with those from PSO-based ANNs, including conventional PSO neural network (CPSONN) and improved PSO neural network (IPSONN) with passive congregation. The IPSONN algorithm improves PSO convergence by using the selfish herd concept in swarm behavior. Our results show that the PSO-based ANNs performed better than LM-NN. For models run using a single parameter (rainfall) as input, the root mean square error (RMSE) of the testing dataset for IPSONN was the lowest (0.152 m) compared to those for CPSONN (0.161 m) and LM-NN (0.205 m). For multi-parameter (rainfall and water level) inputs, the RMSE of the testing dataset for IPSONN was also the lowest (0.089 m) compared to those for CPSONN (0.105 m) and LM-NN (0.145 m). The results also indicate that the LM-NN model performed poorly in predicting the low and peak water levels, in comparison to the PSO-based ANNs. Moreover, the IPSONN model was superior to CPSONN in predicting extreme water levels. Lastly, IPSONN had a quicker convergence rate compared to CPSONN.

Comparison between response surface models and artificial neural networks in hydrologic forecasting

Developing an efficient and accurate hydrologic forecasting model is crucial to managing water resources and flooding issues. In this study, response surface (RS) models including multiple linear regression (MLR), quadratic response surface (QRS), and nonlinear response surface (NRS) were applied to daily runoff (e.g., discharge and water level) prediction. Two catchments, one in southeast China and the other in western Canada, were used to demonstrate the applicability of the proposed models. Their performances were compared with artificial neural network (ANN) models, trained with the learning algorithms of the gradient descent with adaptive learning rate (ANN-GDA) and Levenberg-Marquardt (ANN-LM). The performances of both RS and ANN in relation to the lags used in the input data, the length of the training samples, long-term (monthly and yearly) predictions, and peak value predictions were also analyzed. The results indicate that the QRS and NRS were able to obtain equally good performance in runoff prediction, as compared with ANN-GDA and ANN-LM, but require lower computational efforts. The RS models bring practical benefits in their application to hydrologic forecasting, particularly in the cases of short-term flood forecasting (e.g., hourly) due to fast training capability, and could be considered as an alternative to ANN

Global optimisation of neural networks using a deterministic hybrid approach

Selection of the topology of a neural network and correct parameters for the learning algorithm is a tedious task for designing an optimal artificial neural network, which is smaller, faster and with a better generalization performance. In this paper we introduce a recently developed cutting angle method (a deterministic technique) for global optimization of connection weights. Neural networks are initially trained using the cutting angle method and later the learning is fine-tuned (meta-learning) using conventional gradient descent or other optimization techniques. Experiments were carried out on three time series benchmarks and a comparison was done using evolutionary neural networks. Our preliminary experimentation results show that the proposed deterministic approach could provide near optimal results much faster than the evolutionary approach.

The prediction of damage condition in regards to damage factor influence of light structures on expansive soils in Victoria, Australia

This paper proposes a neural network model using genetic algorithm for a model for the prediction of the damage condition of existing light structures founded in expansive soils in Victoria, Australia. It also accounts for both individual effects and interactive effects of the damage factors influencing the deterioration of light structures. A Neural Network Model was chosen because it can deal with 'noisy' data while a Genetic Algorithm was chosen because it does not get `trapped' in local optimum like other gradient descent methods. The results obtained were promising and indicate that a Neural Network Model trained using a Genetic Algorithm has the ability to develop an interactive relationship and a Predicted Damage Conditions Model.

A hybrid neural network model for noisy data regression

A hybrid neural network model, based on the fusion of fuzzy adaptive resonance theory (FA ART) and the general regression neural network (GRNN), is proposed in this paper. Both FA and the GRNN are incremental learning systems and are very fast in network training. The proposed hybrid model, denoted as GRNNFA, is able to retain these advantages and, at the same time, to reduce the computational requirements in calculating and storing information of the kernels. A clustering version of the GRNN is designed with data compression by FA for noise removal. An adaptive gradient-based kernel width optimization algorithm has also been devised. Convergence of the gradient descent algorithm can be accelerated by the geometric incremental growth of the updating factor. A series of experiments with four benchmark datasets have been conducted to assess and compare effectiveness of GRNNFA with other approaches. The GRNNFA model is also employed in a novel application task for predicting the evacuation time of patrons at typical karaoke centers in Hong Kong in the event of fire. The results positively demonstrate the applicability of GRNNFA in noisy data regression problems.

Fusion of GRNN and FA for online noisy data regression

A new online neural-network-based regression model for noisy data is proposed in this paper. It is a hybrid system combining the Fuzzy ART (FA) and General Regression Neural Network (GRNN) models. Both the FA and GRNN models are fast incremental learning systems. The proposed hybrid model, denoted as GRNNFA-online, retains the online learning properties of both models. The kernel centers of the GRNN are obtained by compressing the training samples using the FA model. The width of each kernel is then estimated by the K-nearest-neighbors (kNN) method. A heuristic is proposed to tune the value of Kof the kNN dynamically based on the concept of gradient-descent. The performance of the GRNNFA-online model was evaluated using two benchmark datasets, i.e., OZONE and Friedman#1. The experimental results demonstrated the convergence of the prediction errors. Bootstrapping was employed to assess the performance statistically. The final prediction errors are analyzed and compared with those from other systems.Bootstrapping was employed to assess the performance statistically. The final prediction errors are analyzed and compared with those from other systems.

Classification of healthcare data using genetic fuzzy logic system and wavelets

Healthcare plays an important role in promoting the general health and well-being of people around the world. The difficulty in healthcare data classification arises from the uncertainty and the high-dimensional nature of the medical data collected. This paper proposes an integration of fuzzy standard additive model (SAM) with genetic algorithm (GA), called GSAM, to deal with uncertainty and computational challenges. GSAM learning process comprises three continual steps: rule initialization by unsupervised learning using the adaptive vector quantization clustering, evolutionary rule optimization by GA and parameter tuning by the gradient descent supervised learning. Wavelet transformation is employed to extract discriminative features for high-dimensional datasets. GSAM becomes highly capable when deployed with small number of wavelet features as its computational burden is remarkably reduced. The proposed method is evaluated using two frequently-used medical datasets: the Wisconsin breast cancer and Cleveland heart disease from the UCI Repository for machine learning. Experiments are organized with a five-fold cross validation and performance of classification techniques are measured by a number of important metrics: accuracy, F-measure, mutual information and area under the receiver operating characteristic curve. Results demonstrate the superiority of the GSAM compared to other machine learning methods including probabilistic neural network, support vector machine, fuzzy ARTMAP, and adaptive neuro-fuzzy inference system. The proposed approach is thus helpful as a decision support system for medical practitioners in the healthcare practice.

Adaptive intelligent energy management system of plug-in hybrid electric vehicle

Efficient energy management in hybrid vehicles is the key for reducing fuel consumption and emissions. To capitalize on the benefits of using PHEVs (Plug-in Hybrid Electric Vehicles), an intelligent energy management system is developed and evaluated in this paper. Models of vehicle engine, air conditioning, powertrain, and hybrid electric drive system are first developed. The effect of road parameters such as bend direction and road slope angle as well as environmental factors such as wind (direction and speed) and thermal conditions are also modeled. Due to the nonlinear and complex nature of the interactions between PHEV-Environment-Driver components, a soft computing based intelligent management system is developed using three fuzzy logic controllers. The crucial fuzzy engine controller within the intelligent energy management system is made adaptive by using a hybrid multi-layer adaptive neuro-fuzzy inference system with genetic algorithm optimization. For adaptive learning, a number of datasets were created for different road conditions and a hybrid learning algorithm based on the least squared error estimate using the gradient descent method was proposed. The proposed adaptive intelligent energy management system can learn while it is running and makes proper adjustments during its operation. It is shown that the proposed intelligent energy management system is improving the performance of other existing systems. © 2014 Elsevier Ltd.

Sparse adaptive multi-hyperplane machine

The Adaptive Multiple-hyperplane Machine (AMM) was recently proposed to deal with large-scale datasets. However, it has no principle to tune the complexity and sparsity levels of the solution. Addressing the sparsity is important to improve learning generalization, prediction accuracy and computational speedup. In this paper, we employ the max-margin principle and sparse approach to propose a new Sparse AMM (SAMM). We solve the new optimization objective function with stochastic gradient descent (SGD). Besides inheriting the good features of SGD-based learning method and the original AMM, our proposed Sparse AMM provides machinery and flexibility to tune the complexity and sparsity of the solution, making it possible to avoid overfitting and underfitting. We validate our approach on several large benchmark datasets. We show that with the ability to control sparsity, the proposed Sparse AMM yields superior classification accuracy to the original AMM while simultaneously achieving computational speedup.

Parallelization of the discrete gradient method of non-smooth optimization and its applications

We investigate parallelization and performance of the discrete gradient method of nonsmooth optimization. This derivative free method is shown to be an effective optimization tool, able to skip many shallow local minima of nonconvex nondifferentiable objective functions. Although this is a sequential iterative method, we were able to parallelize critical steps of the algorithm, and this lead to a significant improvement in performance on multiprocessor computer clusters. We applied this method to a difficult polyatomic clusters problem in computational chemistry, and found this method to outperform other algorithms.

Frequency-domain beamformers using conjugate gradient techniques for speech enhancement

A multiple-iteration constrained conjugate gradient (MICCG) algorithm and a single-iteration constrained conjugate gradient (SICCG) algorithm are proposed to realize the widely used frequency-domain minimum-variance-distortionless-response (MVDR) beamformers and the resulting algorithms are applied to speech enhancement. The algorithms are derived based on the Lagrange method and the conjugate gradient techniques. The implementations of the algorithms avoid any form of explicit or implicit autocorrelation matrix inversion. Theoretical analysis establishes formal convergence of the algorithms. Specifically, the MICCG algorithm is developed based on a block adaptation approach and it generates a finite sequence of estimates that converge to the MVDR solution. For limited data records, the estimates of the MICCG algorithm are better than the conventional estimators and equivalent to the auxiliary vector algorithms. The SICCG algorithm is developed based on a continuous adaptation approach with a sample-by-sample updating procedure and the estimates asymptotically converge to the MVDR solution. An illustrative example using synthetic data from a uniform linear array is studied and an evaluation on real data recorded by an acoustic vector sensor array is demonstrated. Performance of the MICCG algorithm and the SICCG algorithm are compared with the state-of-the-art approaches.