Mixture density networks

Minimization of a sum-of-squares or cross-entropy error function leads to network outputs which approximate the conditional averages of the target data, conditioned on the input vector. For classifications problems, with a suitably chosen target coding scheme, these averages represent the posterior probabilities of class membership, and so can be regarded as optimal. For problems involving the prediction of continuous variables, however, the conditional averages provide only a very limited description of the properties of the target variables. This is particularly true for problems in which the mapping to be learned is multi-valued, as often arises in the solution of inverse problems, since the average of several correct target values is not necessarily itself a correct value. In order to obtain a complete description of the data, for the purposes of predicting the outputs corresponding to new input vectors, we must model the conditional probability distribution of the target data, again conditioned on the input vector. In this paper we introduce a new class of network models obtained by combining a conventional neural network with a mixture density model. The complete system is called a Mixture Density Network, and can in principle represent arbitrary conditional probability distributions in the same way that a conventional neural network can represent arbitrary functions. We demonstrate the effectiveness of Mixture Density Networks using both a toy problem and a problem involving robot inverse kinematics.

On the relationship between Bayesian error bars and the input data density

We investigate the dependence of Bayesian error bars on the distribution of data in input space. For generalized linear regression models we derive an upper bound on the error bars which shows that, in the neighbourhood of the data points, the error bars are substantially reduced from their prior values. For regions of high data density we also show that the contribution to the output variance due to the uncertainty in the weights can exhibit an approximate inverse proportionality to the probability density. Empirical results support these conclusions.

Laminar distribution of neurofilament inclusions and swollen achromatic neurons in neurofilament inclusion disease (NID)

The laminar distribution of the neurofilament inclusions (NI) and swollen achromatic neurons (SN) was studied in gyri of the temporal cortex in four patients with neurofilament inclusion disease (NID). In 84% of gyri analysed, the density of the NI was maximal in the lower cortical laminae. The distribution of the SN was more variable than the NI. Density was maximal in the lower cortex in 46% of gyri, in the upper cortical laminae in 8% of gyri, and a bimodal distribution in 15% of gyri. In the remaining gyri, there was a more even distribution of SN with cortical depth. In 31% of gyri, the vertical density of the NI was positively correlated with that of the SN. The data suggest that cortical degeneration in the temporal lobe of NID initially affects neurons in the lower laminae. Subsequently, the pathology may spread to affect much of the cortical profile, the SN preceding the appearance of the NI.

Regularisation of mixture density networks

Mixture Density Networks are a principled method to model conditional probability density functions which are non-Gaussian. This is achieved by modelling the conditional distribution for each pattern with a Gaussian Mixture Model for which the parameters are generated by a neural network. This thesis presents a novel method to introduce regularisation in this context for the special case where the mean and variance of the spherical Gaussian Kernels in the mixtures are fixed to predetermined values. Guidelines for how these parameters can be initialised are given, and it is shown how to apply the evidence framework to mixture density networks to achieve regularisation. This also provides an objective stopping criteria that can replace the `early stopping' methods that have previously been used. If the neural network used is an RBF network with fixed centres this opens up new opportunities for improved initialisation of the network weights, which are exploited to start training relatively close to the optimum. The new method is demonstrated on two data sets. The first is a simple synthetic data set while the second is a real life data set, namely satellite scatterometer data used to infer the wind speed and wind direction near the ocean surface. For both data sets the regularisation method performs well in comparison with earlier published results. Ideas on how the constraint on the kernels may be relaxed to allow fully adaptable kernels are presented.

Density evolution for multistage CDMA multiuser detector

An analytical framework to analyze the stage-by-stage detection dynamics of the multistage CDMA multiuser detector is presented. The density evolution idea is applied to analyze the multistage detector. Message distribution is treated basically by a Gaussian approximation, but interstage correlation of messages is systematically taken into account, which turns out to provide significant improvement.

Multi-valued control problems and mixture density network

We have proposed a novel robust inversion-based neurocontroller that searches for the optimal control law by sampling from the estimated Gaussian distribution of the inverse plant model. However, for problems involving the prediction of continuous variables, a Gaussian model approximation provides only a very limited description of the properties of the inverse model. This is usually the case for problems in which the mapping to be learned is multi-valued or involves hysteritic transfer characteristics. This often arises in the solution of inverse plant models. In order to obtain a complete description of the inverse model, a more general multicomponent distributions must be modeled. In this paper we test whether our proposed sampling approach can be used when considering an arbitrary conditional probability distributions. These arbitrary distributions will be modeled by a mixture density network. Importance sampling provides a structured and principled approach to constrain the complexity of the search space for the ideal control law. The effectiveness of the importance sampling from an arbitrary conditional probability distribution will be demonstrated using a simple single input single output static nonlinear system with hysteretic characteristics in the inverse plant model.

Laminar distribution of the pathological changes in frontal and temporal cortex in 8 patients with progressive supranuclear palsy

OBJECTIVE: To determine the laminar distribution of the pathological changes in the cerebral cortex in progressive supranuclear palsy (PSP). METHOD: The distribution of the abnormally enlarged neurons (EN), surviving neurons, neurofibrillary tangles (NFT), glial inclusions (GI), tufted astrocytes (TA), and neuritic plaques (NP) were studied across the cortex in tau immunolabeled sections of frontal and temporal cortex in 8 cases of PSP. RESULTS: The distribution of the NFT was highly variable with no consistent pattern of laminar distribution. The GI were distributed either in the lower laminae or uniformly across the cortex. Surviving neurons exhibited either a density peak in the upper laminae or a bimodal distribution was present with density peaks in the upper and lower laminae. The EN and glial cell nuclei were distributed primarily in the lower cortical laminae. There were positive correlations between the densities of the EN and glial cell nuclei and negative correlations between the surviving neurons and glial cells. No correlations were present between the densities of the NFT and GI. CONCLUSION: Cortical pathology in PSP predominantly affects the lower laminae but may spread to affect the upper laminae in some cases. The NFT and GI may have different laminar distributions and gliosis occurs concurrently with neuronal enlargement.

Laminar distribution of β-amyloid deposits in dementia with Lewy bodies and in Alzheimer's disease

This study tested whether the laminar distribution of the β-amyloid (Aβ) deposits in dementia with Lewy bodies (DLB) cases with significant Alzheimer's disease (AD) pathology (DLB/AD) was similar to "pure" AD. In DLB/AD, the maximum density of the diffuse and primitive deposits occurred either in the upper laminae or a bimodal distribution was present with density peaks in the upper and lower laminae. A bimodal distribution of the classic Aβ deposits was also observed. Compared with AD, DLB/AD cases had fewer primitive deposits relative to the diffuse and classic deposits; the primitive deposits exhibited a bimodal distribution more frequently, and the diffuse deposits occurred more often in the upper laminae. These results suggest that Aβ pathology in DLB/AD may not simply represent the presence of associated AD. © 2006 Sage Publications.

Multiple system atrophy: laminar distribution of the pathological changes in frontal and temporal neocortex--a study in ten patients

OBJECTIVE: To determine the distribution of the pathological changes in the neocortex in multiple-system atrophy (MSA). METHOD: The vertical distribution of the abnormal neurons (neurons with enlarged or atrophic perikarya), surviving neurons, glial cytoplasmic inclusions (GCI) and neuronal cytoplasmic inclusions (NI) were studied in alpha-synuclein-stained material of frontal and temporal cortex in ten cases of MSA. RESULTS: Abnormal neurons exhibited two common patterns of distribution, viz., density was either maximal in the upper cortex or a bimodal distribution was present with a density peak in the upper and lower cortex. The NI were either located in the lower cortex or were more uniformly distributed down the cortical profile. The distribution of the GCI varied considerably between gyri and cases. The density of the glial cell nuclei was maximal in the lower cortex in the majority of gyri. In a number of gyri, there was a positive correlation between the vertical densities of the abnormal neurons, the total number of surviving neurons, and the glial cell nuclei. The vertical densities of the GCI were not correlated with those of the surviving neurons or glial cells but the GCI and NI were positively correlated in a small number of gyri. CONCLUSION: The data suggest that there is significant degeneration of the frontal and temporal lobes in MSA, the lower laminae being affected more significantly than the upper laminae. Cortical degeneration in MSA is likely to be secondary to pathological changes occurring within subcortical areas.

The distribution of senile plaques, neurofibrillary tangles and beta/A4 deposits in the hippocampus in Alzheimer's disease

The density of senile plaques (SP) and neurofibrillary tangles (NFT) was studied in Glees and Marsland stained sections of the hippocampus and parahippocampal gyrus (PHG) in 20 pateints with Alzheimer's disease. In addition, in six of the patients, the density of beta/A4 protein deposits, as revealed by immunohistochemistry and neurofibrillary changes demonstrated with the Gallyas stain, were studied in adjacent sections. The density of Glees SP and beta/A4 deposits was significantly greater in area CA1 of the hippocampus and in the subiculum than in the PHG. Hence, neurofibrillary degeneration appears to be a more important lesion than beta/A4 deposition in the hippocampus compared with the PHG. In addition, the detailed distribution of the lesions in the hippocampus could be explained if beta/A4/SP and NFT occur on the axon terminals and in the cell bodies respectively of the same neurons.

Regularisation of mixture density networks

Mixture Density Networks are a principled method to model conditional probability density functions which are non-Gaussian. This is achieved by modelling the conditional distribution for each pattern with a Gaussian Mixture Model for which the parameters are generated by a neural network. This thesis presents a novel method to introduce regularisation in this context for the special case where the mean and variance of the spherical Gaussian Kernels in the mixtures are fixed to predetermined values. Guidelines for how these parameters can be initialised are given, and it is shown how to apply the evidence framework to mixture density networks to achieve regularisation. This also provides an objective stopping criteria that can replace the `early stopping' methods that have previously been used. If the neural network used is an RBF network with fixed centres this opens up new opportunities for improved initialisation of the network weights, which are exploited to start training relatively close to the optimum. The new method is demonstrated on two data sets. The first is a simple synthetic data set while the second is a real life data set, namely satellite scatterometer data used to infer the wind speed and wind direction near the ocean surface. For both data sets the regularisation method performs well in comparison with earlier published results. Ideas on how the constraint on the kernels may be relaxed to allow fully adaptable kernels are presented.

Laminar distribution of the pathological changes in the cerebral cortex in variant Creutzfeldt-Jakob disease (vCJD)

To determine the pattern of cortical degeneration in cases of variant Creutzfeldt-Jakob disease (vCJD), the laminar distribution of the vacuolation ("spongiform change"), surviving neurones, glial cell nuclei, and prion protein (PrP) deposits was studied in the frontal, parietal and temporal lobes. The vacuolation exhibited two common patterns of distribution: either the vacuoles were present throughout the cortex or a bimodal distribution was present with peaks of density in the upper and lower cortical laminae. The distribution of the surviving neurones was highly variable in different regions; the commonest pattern being a uniform distribution with cortical depth. Glial cell nuclei were distributed largely in the lower cortical laminae. The non-florid PrP deposits exhibited either a bimodal distribution or exhibited a peak of density in the upper cortex while the florid deposits were either uniformly distributed down the cortex or were present in the upper cortical laminae. In a significant proportion of areas, the density of the vacuoles was positively correlated with either the surviving neurones or with the glial cell nuclei. These results suggest similarities and differences in the laminar distributions of the pathogenic changes in vCJD compared with cases of sporadic CJD (sCJD). The laminar distribution of vacuoles was more extensive in vCJD than in sCJD whereas the distribution of the glial cell nuclei was similar in the two disorders. In addition, PrP deposits in sCJD were localised mainly in the lower cortical laminae while in vCJD, PrP deposits were either present in all laminae or restricted to the upper cortical laminae. These patterns of laminar distribution suggest that the process of cortical degeneration may be distinctly different in vCJD compared with sCJD.

Laminar distribution of Pick bodies, Pick cells and Alzheimer disease pathology in the frontal and temporal cortex in Pick's disease

Lesions in Alzheimer's disease (AD) and dementia with Lewy bodies (DLB) have distinct laminar distributions in the cortex. The objective of the present study was to test the hypothesis that the lesions characteristic of Pick's disease (PD) and AD have distinctly different laminar distributions in cases of PD. Hence, the laminar distribution of Pick bodies (PB), Pick cells (PC), senile plaques (SP) and neurofibrillary tangles (NFT) was studied in the frontal and temporal cortex in nine patients with PD. In 57% of analyses of individual cortical areas, the density of PB was maximal in the upper cortex while in 25% of analyses, the distribution of PB was bimodal with density peaks in the upper and lower cortex. The density of PC was maximal in the lower cortex in 77% of analyses while a bimodal distribution was present in 5% of analyses. The density of NFT was maximal in the upper cortex in 50% of analyses, in the lower cortex in 15% of analyses, with a bimodal distribution in 4% of analyses. The density of SP did not vary significantly with cortical depth in 86% of analyses. The vertical densities of PB and PC were negatively correlated in 12/21 (57%) of brain areas. The maximum density of PB in the upper cortex was positively correlated with the maximum density of PC in the lower cortex. In 17/25 (68%) of brain areas, there was no significant correlation between the vertical densities of PB and NFT. The data suggest that the pathogenesis of PB may be related to that of the PC. In addition, although in many areas PB and NFT occur predominantly in the upper cortex, the two lesions appeared to affect different neuronal populations.

Laminar distribution of amyloid-beta (Abeta) deposits in dementia with Lewy bodies: comparison with Alzheimer's disease

Significant amyloid-beta (Abeta) deposition in cases of dementia with Lewy bodies (DLB) may represent concurrent Alzheimer's disease (AD). To test this hypothesis, the laminar distribution of the diffuse, primitive, and classic Abeta deposits was studied in the frontal and temporal cortex in cases of DLB and were compared with AD. In DLB, the diffuse and primitive deposits exhibited two common patterns of distribution; either maximum density occurred in the upper cortical laminae or a bimodal distribution was present with density peaks in the upper and lower laminae. In addition, a bimodal distribution of the classic deposits was observed in approximately half of the cortical areas analysed. A number of differences in the laminar distributions of Abeta deposits were observed in DLB and AD. First, the proportion of the primitive relative to the diffuse and classic deposits present was lower in DLB compared with AD. Second, the primitive deposits were more frequently bimodally distributed in DLB. Third, the density of the diffuse deposits reached a maximum lower in the cortical profile in AD. These data suggest differences in the pattern of cortical degeneration in the two disorders and therefore, DLB cases with significant Abeta pathology may not represent the coexistence of DLB and AD.

Laminar distribution of ballooned neurons and tau positive neurons with inclusions in patients with corticobasal degeneration

The laminar distribution of ballooned neurons (BN) and tau positive neurons with inclusions (tau+ neurons) was studied in the frontal and temporal cortex in twelve patients with corticobasal degeneration (CBD). In the majority of brain areas, the density of BN and tau+ neurons was maximal in the lower and upper cortical laminae respectively. The densities of tau+ neurons in the upper and lower cortex were positively correlated. In the majority of brain areas, however, no correlations were observed between the densities of BN and tau+ neurons. The laminar distribution of the BN may reflect the degeneration of the feedback cortico-cortical and/or the efferent cortical pathways. By contrast, the distribution of the tau+ neurons may reflect the degeneration of the feed-forward cortico-cortical pathways. In addition, BN and tau+ neurons may arise as a result of distinct pathological processes.