Bayesian training of mixture density networks

Mixture Density Networks (MDNs) are a well-established method for modelling the conditional probability density which is useful for complex multi-valued functions where regression methods (such as MLPs) fail. In this paper we extend earlier research of a regularisation method for a special case of MDNs to the general case using evidence based regularisation and we show how the Hessian of the MDN error function can be evaluated using R-propagation. The method is tested on two data sets and compared with early stopping.

Mixture density network training by computation in parameter space

Training Mixture Density Network (MDN) configurations within the NETLAB framework takes time due to the nature of the computation of the error function and the gradient of the error function. By optimising the computation of these functions, so that gradient information is computed in parameter space, training time is decreased by at least a factor of sixty for the example given. Decreased training time increases the spectrum of problems to which MDNs can be practically applied making the MDN framework an attractive method to the applied problem solver.

EM optimization of latent-variable density models

There is currently considerable interest in developing general non-linear density models based on latent, or hidden, variables. Such models have the ability to discover the presence of a relatively small number of underlying `causes' which, acting in combination, give rise to the apparent complexity of the observed data set. Unfortunately, to train such models generally requires large computational effort. In this paper we introduce a novel latent variable algorithm which retains the general non-linear capabilities of previous models but which uses a training procedure based on the EM algorithm. We demonstrate the performance of the model on a toy problem and on data from flow diagnostics for a multi-phase oil pipeline.

Regularisation of mixture density networks

Mixture Density Networks are a principled method to model conditional probability density functions which are non-Gaussian. This is achieved by modelling the conditional distribution for each pattern with a Gaussian Mixture Model for which the parameters are generated by a neural network. This thesis presents a novel method to introduce regularisation in this context for the special case where the mean and variance of the spherical Gaussian Kernels in the mixtures are fixed to predetermined values. Guidelines for how these parameters can be initialised are given, and it is shown how to apply the evidence framework to mixture density networks to achieve regularisation. This also provides an objective stopping criteria that can replace the `early stopping' methods that have previously been used. If the neural network used is an RBF network with fixed centres this opens up new opportunities for improved initialisation of the network weights, which are exploited to start training relatively close to the optimum. The new method is demonstrated on two data sets. The first is a simple synthetic data set while the second is a real life data set, namely satellite scatterometer data used to infer the wind speed and wind direction near the ocean surface. For both data sets the regularisation method performs well in comparison with earlier published results. Ideas on how the constraint on the kernels may be relaxed to allow fully adaptable kernels are presented.

Regularisation of mixture density networks

Mixture Density Networks are a principled method to model conditional probability density functions which are non-Gaussian. This is achieved by modelling the conditional distribution for each pattern with a Gaussian Mixture Model for which the parameters are generated by a neural network. This thesis presents a novel method to introduce regularisation in this context for the special case where the mean and variance of the spherical Gaussian Kernels in the mixtures are fixed to predetermined values. Guidelines for how these parameters can be initialised are given, and it is shown how to apply the evidence framework to mixture density networks to achieve regularisation. This also provides an objective stopping criteria that can replace the `early stopping' methods that have previously been used. If the neural network used is an RBF network with fixed centres this opens up new opportunities for improved initialisation of the network weights, which are exploited to start training relatively close to the optimum. The new method is demonstrated on two data sets. The first is a simple synthetic data set while the second is a real life data set, namely satellite scatterometer data used to infer the wind speed and wind direction near the ocean surface. For both data sets the regularisation method performs well in comparison with earlier published results. Ideas on how the constraint on the kernels may be relaxed to allow fully adaptable kernels are presented.

Whole body vibration treatments in postmenopausal women can improve bone mineral density:results of a stimulus focussed meta-analysis

Whole body vibration treatment is a non-pharmacological intervention intended to stimulate muscular response and increase bone mineral density, particularly for postmenopausal women. The literature related to this topic is controversial, heterogeneous, and unclear despite the prospect of a major clinical effect. The aim of this study was to identify and systematically review the literature to assess the effect of whole body vibration treatments on bone mineral density (BMD) in postmenopausal women with a specific focus on the experimental factors that influence the stimulus. Nine studies fulfilled the inclusion criteria, including 527 postmenopausal women and different vibration delivery designs. Cumulative dose, amplitudes and frequency of treatments as well as subject posture during treatment vary widely among studies. Some of the studies included an associated exercise training regime. Both randomized and controlled clinical trials were included. Whole body vibration was shown to produce significant BMD improvements on the hip and spine when compared to no intervention. Conversely, treatment associated with exercise training resulted in negligible outcomes when compared to exercise training or to placebo. Moreover, side-alternating platforms were more effective in improving BMD values than synchronous platforms and mechanical oscillations of magnitude higher than 3 g and/or frequency lower than 25 Hz were also found to be effective. Treatments with a cumulative dose over 1000 minutes in the follow-up period were correlated to positive outcomes. Our conclusion is that whole body vibration treatments in elderly women can reduce BMD decline.However, many factors (e.g. amplitude, frequency and subject posture) affect the capacity of the vibrations to propagate to the target site; the adequate level of stimulation required to produce these effects has not yet been defined. Further biomechanical analyses to predict the propagation of the vibration waves along the body and assess the stimulation levels are required.