Self-organized phase transitions in neural networks as a neural mechanism of information processing.

Transitions between dynamically stable activity patterns imposed on an associative neural network are shown to be induced by self-organized infinitesimal changes in synaptic connection strength and to be a kind of phase transition. A key event for the neural process of information processing in a population coding scheme is transition between the activity patterns encoding usual entities. We propose that the infinitesimal and short-term synaptic changes based on the Hebbian learning rule are the driving force for the transition. The phase transition between the following two dynamical stable states is studied in detail, the state where the firing pattern is changed temporally so as to itinerate among several patterns and the state where the firing pattern is fixed to one of several patterns. The phase transition from the pattern itinerant state to a pattern fixed state may be induced by the Hebbian learning process under a weak input relevant to the fixed pattern. The reverse transition may be induced by the Hebbian unlearning process without input. The former transition is considered as recognition of the input stimulus, while the latter is considered as clearing of the used input data to get ready for new input. To ensure that information processing based on the phase transition can be made by the infinitesimal and short-term synaptic changes, it is absolutely necessary that the network always stays near the critical state corresponding to the phase transition point.

Chimeric plant calcium/calmodulin-dependent protein kinase gene with a neural visinin-like calcium-binding domain.

Calcium, a universal second messenger, regulates diverse cellular processes in eukaryotes. Ca2+ and Ca2+/calmodulin-regulated protein phosphorylation play a pivotal role in amplifying and diversifying the action of Ca(2+)-binding domain was cloned and characterized from lily. The cDNA clone contains an open reading frame coding for a protein of 520 amino acids. The predicted structure of CCaMK contains a catalytic domain followed by two regulatory domains, a calmodulin-binding domain and a visinin-like Ca(2+)-binding domain. The amino-terminal region of CCaMK contains all 11 conserved subdomains characteristic of serine/threonine protein kinases. The calmodulin-binding region of CCaMK has high homology (79%) to alpha subunit of mammalian Ca2+/calmodulin-dependent protein kinase. The calmodulin-binding region is fused to a neural visinin-like domain that contains three Ca(2+)-binding EF-hand motifs and a biotin-binding site. The Escherichia coli-expressed protein (approximately 56 kDa) binds calmodulin in a Ca(2+)-dependent manner. Furthermore, 45Ca-binding assays revealed that CCaMK directly binds Ca2+. The CCaMK gene is preferentially expressed in developing anthers. Southern blot analysis revealed that CCaMK is encoded by a single gene. The structural features of the gene suggest that it has multiple regulatory controls and could play a unique role in Ca2+ signaling in plants.

Non-coding RNAs in the nervous system

Increasing evidence suggests that the development and function of the nervous system is heavily dependent on RNA editing and the intricate spatiotemporal expression of a wide repertoire of non-coding RNAs, including micro RNAs, small nucleolar RNAs and longer non-coding RNAs. Non-coding RNAs may provide the key to understanding the multi-tiered links between neural development, nervous system function, and neurological diseases.

Sparse representation for Gaussian process models

We develop an approach for a sparse representation for Gaussian Process (GP) models in order to overcome the limitations of GPs caused by large data sets. The method is based on a combination of a Bayesian online algorithm together with a sequential construction of a relevant subsample of the data which fully specifies the prediction of the model. Experimental results on toy examples and large real-world datasets indicate the efficiency of the approach.

Message passing for task redistribution on sparse graphs

The problem of resource allocation in sparse graphs with real variables is studied using methods of statistical physics. An efficient distributed algorithm is devised on the basis of insight gained from the analysis and is examined using numerical simulations, showing excellent performance and full agreement with the theoretical results.

Sparse on-line Gaussian processes

We develop an approach for sparse representations of Gaussian Process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations for large data sets. The method is based on a combination of a Bayesian online algorithm together with a sequential construction of a relevant subsample of the data which fully specifies the prediction of the GP model. By using an appealing parametrisation and projection techniques that use the RKHS norm, recursions for the effective parameters and a sparse Gaussian approximation of the posterior process are obtained. This allows both for a propagation of predictions as well as of Bayesian error measures. The significance and robustness of our approach is demonstrated on a variety of experiments.

The role of stochasticity in an information-optimal neural population code

In this paper we consider the optimisation of Shannon mutual information (MI) in the context of two model neural systems The first is a stochastic pooling network (population) of McCulloch-Pitts (MP) type neurons (logical threshold units) subject to stochastic forcing; the second is (in a rate coding paradigm) a population of neurons that each displays Poisson statistics (the so called 'Poisson neuron'). The mutual information is optimised as a function of a parameter that characterises the 'noise level'-in the MP array this parameter is the standard deviation of the noise, in the population of Poisson neurons it is the window length used to determine the spike count. In both systems we find that the emergent neural architecture and; hence, code that maximises the MI is strongly influenced by the noise level. Low noise levels leads to a heterogeneous distribution of neural parameters (diversity), whereas, medium to high noise levels result in the clustering of neural parameters into distinct groups that can be interpreted as subpopulations In both cases the number of subpopulations increases with a decrease in noise level. Our results suggest that subpopulations are a generic feature of an information optimal neural population.

Evolutionary neural networks : models and applications

The scaling problems which afflict attempts to optimise neural networks (NNs) with genetic algorithms (GAs) are disclosed. A novel GA-NN hybrid is introduced, based on the bumptree, a little-used connectionist model. As well as being computationally efficient, the bumptree is shown to be more amenable to genetic coding lthan other NN models. A hierarchical genetic coding scheme is developed for the bumptree and shown to have low redundancy, as well as being complete and closed with respect to the search space. When applied to optimising bumptree architectures for classification problems the GA discovers bumptrees which significantly out-perform those constructed using a standard algorithm. The fields of artificial life, control and robotics are identified as likely application areas for the evolutionary optimisation of NNs. An artificial life case-study is presented and discussed. Experiments are reported which show that the GA-bumptree is able to learn simulated pole balancing and car parking tasks using only limited environmental feedback. A simple modification of the fitness function allows the GA-bumptree to learn mappings which are multi-modal, such as robot arm inverse kinematics. The dynamics of the 'geographic speciation' selection model used by the GA-bumptree are investigated empirically and the convergence profile is introduced as an analytical tool. The relationships between the rate of genetic convergence and the phenomena of speciation, genetic drift and punctuated equilibrium arc discussed. The importance of genetic linkage to GA design is discussed and two new recombination operators arc introduced. The first, linkage mapped crossover (LMX) is shown to be a generalisation of existing crossover operators. LMX provides a new framework for incorporating prior knowledge into GAs.Its adaptive form, ALMX, is shown to be able to infer linkage relationships automatically during genetic search.

Colour image coding with wavelets and matching pursuit

This thesis considers sparse approximation of still images as the basis of a lossy compression system. The Matching Pursuit (MP) algorithm is presented as a method particularly suited for application in lossy scalable image coding. Its multichannel extension, capable of exploiting inter-channel correlations, is found to be an efficient way to represent colour data in RGB colour space. Known problems with MP, high computational complexity of encoding and dictionary design, are tackled by finding an appropriate partitioning of an image. The idea of performing MP in the spatio-frequency domain after transform such as Discrete Wavelet Transform (DWT) is explored. The main challenge, though, is to encode the image representation obtained after MP into a bit-stream. Novel approaches for encoding the atomic decomposition of a signal and colour amplitudes quantisation are proposed and evaluated. The image codec that has been built is capable of competing with scalable coders such as JPEG 2000 and SPIHT in terms of compression ratio.

Sparse image representation with encryption

In this thesis we present an overview of sparse approximations of grey level images. The sparse representations are realized by classic, Matching Pursuit (MP) based, greedy selection strategies. One such technique, termed Orthogonal Matching Pursuit (OMP), is shown to be suitable for producing sparse approximations of images, if they are processed in small blocks. When the blocks are enlarged, the proposed Self Projected Matching Pursuit (SPMP) algorithm, successfully renders equivalent results to OMP. A simple coding algorithm is then proposed to store these sparse approximations. This is shown, under certain conditions, to be competitive with JPEG2000 image compression standard. An application termed image folding, which partially secures the approximated images is then proposed. This is extended to produce a self contained folded image, containing all the information required to perform image recovery. Finally a modified OMP selection technique is applied to produce sparse approximations of Red Green Blue (RGB) images. These RGB approximations are then folded with the self contained approach.

A hierarchy of phase transitions in optimal neuronal coding:from binary to M-ary discrete optimal codes

We have investigated how optimal coding for neural systems changes with the time available for decoding. Optimization was in terms of maximizing information transmission. We have estimated the parameters for Poisson neurons that optimize Shannon transinformation with the assumption of rate coding. We observed a hierarchy of phase transitions from binary coding, for small decoding times, toward discrete (M-ary) coding with two, three and more quantization levels for larger decoding times. We postulate that the presence of subpopulations with specific neural characteristics could be a signiture of an optimal population coding scheme and we use the mammalian auditory system as an example.

Message-passing for inference and optimization of real variables on sparse graphs

The inference and optimization in sparse graphs with real variables is studied using methods of statistical mechanics. Efficient distributed algorithms for the resource allocation problem are devised. Numerical simulations show excellent performance and full agreement with the theoretical results. © Springer-Verlag Berlin Heidelberg 2006.

Searching for Nearest Strings with Neural-Like String Embedding

We analyze an approach to a similarity preserving coding of symbol sequences based on neural distributed representations and show that it can be viewed as a metric embedding process.

A new unsupervised convolutional neural network model for Chinese scene text detection

As one of the most popular deep learning models, convolution neural network (CNN) has achieved huge success in image information extraction. Traditionally CNN is trained by supervised learning method with labeled data and used as a classifier by adding a classification layer in the end. Its capability of extracting image features is largely limited due to the difficulty of setting up a large training dataset. In this paper, we propose a new unsupervised learning CNN model, which uses a so-called convolutional sparse auto-encoder (CSAE) algorithm pre-Train the CNN. Instead of using labeled natural images for CNN training, the CSAE algorithm can be used to train the CNN with unlabeled artificial images, which enables easy expansion of training data and unsupervised learning. The CSAE algorithm is especially designed for extracting complex features from specific objects such as Chinese characters. After the features of articficial images are extracted by the CSAE algorithm, the learned parameters are used to initialize the first CNN convolutional layer, and then the CNN model is fine-Trained by scene image patches with a linear classifier. The new CNN model is applied to Chinese scene text detection and is evaluated with a multilingual image dataset, which labels Chinese, English and numerals texts separately. More than 10% detection precision gain is observed over two CNN models.

The Role of TET Proteins in the Epigenetic Regulation of Neural Gene Expression and Behavior

Understanding how genes affect behavior is critical to develop precise therapies for human behavioral disorders. The ability to investigate the relationship between genes and behavior has been greatly advanced over the last few decades due to progress in gene-targeting technology. Recently, the Tet gene family was discovered and implicated in epigenetic modification of DNA methylation by converting 5-methylcytosine to 5-hydroxymethylcytosine (5hmC). 5hmC and its catalysts, the TET proteins, are highly abundant in the postnatal brain but with unclear functions. To investigate their neural functions, we generated new lines of Tet1 and Tet3 mutant mice using a gene targeting approach. We designed both mutations to cause a frameshift by deleting the largest coding exon of Tet1 (Tet1Δe4) and the catalytic domain of Tet3 (Tet3Δe7-9). As Tet1 is also highly expressed in embryonic stem cells (ESCs), we generated Tet1 homozygous deleted ESCs through sequential targeting to compare the function of Tet1 in the brain to its role in ESCs. To test our hypothesis that TET proteins epigenetically regulate transcription of key neural genes important for normal brain function, we examined transcriptional and epigenetic differences in the Tet1Δe4 mouse brain. The oxytocin receptor (OXTR), a neural gene implicated in social behaviors, is suggested to be epigenetically regulated by an unknown mechanism. Interestingly, several human studies have found associations between OXTR DNA hypermethylation and a wide spectrum of behavioral traits and neuropsychiatric disorders including autism spectrum disorders. Here we report the first evidence for an epigenetic mechanism of Oxtr transcription as expression of Oxtr is reduced in the brains of Tet1Δe4-/- mice. Likewise, the CpG island overlapping the promoter of Oxtr is hypermethylated during early embryonic development and persists into adulthood. We also discovered altered histone modifications at the hypermethylated regions, indicating the loss of TET1 has broad effects on the chromatin structure at Oxtr. Unexpectedly, we discovered an array of novel mRNA isoforms of Oxtr that are selectively reduced in Tet1Δe4-/- mice. Additionally, Tet1Δe4-/- mice display increased agonistic behaviors and impaired maternal care and short-term memory. Our findings support a novel role for TET1 in regulating Oxtr expression by preventing DNA hypermethylation and implicate TET1 in social behaviors, offering novel insight into Oxtr epigenetic regulation and its role in neuropsychiatric disorders.