Mutual Information-based RBM Neural Networks

(Deep) neural networks are increasingly being used for various computer vision and pattern recognition tasks due to their strong ability to learn highly discriminative features. However, quantitative analysis of their classication ability and design philosophies are still nebulous. In this work, we use information theory to analyze the concatenated restricted Boltzmann machines (RBMs) and propose a mutual information-based RBM neural networks (MI-RBM). We develop a novel pretraining algorithm to maximize the mutual information between RBMs. Extensive experimental results on various classication tasks show the eectiveness of the proposed approach.

Calculation of mutual information for nonlinear communication channel at large signal-to-noise ratio

Using the path-integral technique we examine the mutual information for the communication channel modeled by the nonlinear Schrödinger equation with additive Gaussian noise. The nonlinear Schrödinger equation is one of the fundamental models in nonlinear physics, and it has a broad range of applications, including fiber optical communications - the backbone of the internet. At large signal-to-noise ratio we present the mutual information through the path-integral, which is convenient for the perturbative expansion in nonlinearity. In the limit of small noise and small nonlinearity we derive analytically the first nonzero nonlinear correction to the mutual information for the channel.

Conditional mutual inclusive information enables accurate quantification of associations in gene regulatory networks

International audience

Energy and information in Hodgkin-Huxley neurons

[EN]The generation of spikes by neurons is energetically a costly process and the evaluation of the metabolic energy required to maintain the signaling activity of neurons a challenge of practical interest. Neuron models are frequently used to represent the dynamics of real neurons but hardly ever to evaluate the electrochemical energy required to maintain that dynamics. This paper discusses the interpretation of a Hodgkin-Huxley circuit as an energy model for real biological neurons and uses it to evaluate the consumption of metabolic energy in the transmission of information between neurons coupled by electrical synapses, i.e., gap junctions. We show that for a single postsynaptic neuron maximum energy efficiency, measured in bits of mutual information per molecule of adenosine triphosphate (ATP) consumed, requires maximum energy consumption. For groups of parallel postsynaptic neurons we determine values of the synaptic conductance at which the energy efficiency of the transmission presents clear maxima at relatively very low values of metabolic energy consumption. Contrary to what could be expected, the best performance occurs at a low energy cost.

Quantifying synergistic information

Within the microcosm of information theory, I explore what it means for a system to be functionally irreducible. This is operationalized as quantifying the extent to which cooperative or “synergistic” effects enable random variables X₁, ... , X_n to predict (have mutual information about) a single target random variable Y . In Chapter 1, we introduce the problem with some emblematic examples. In Chapter 2, we show how six different measures from the existing literature fail to quantify this notion of synergistic mutual information. In Chapter 3 we take a step towards a measure of synergy which yields the first nontrivial lowerbound on synergistic mutual information. In Chapter 4, we find that synergy is but the weakest notion of a broader concept of irreducibility. In Chapter 5, we apply our results from Chapters 3 and 4 towards grounding Giulio Tononi’s ambitious φ measure which attempts to quantify the magnitude of consciousness experience.

Predictive coding and the slowness principle: an information-theoretic approach.

Understanding the guiding principles of sensory coding strategies is a main goal in computational neuroscience. Among others, the principles of predictive coding and slowness appear to capture aspects of sensory processing. Predictive coding postulates that sensory systems are adapted to the structure of their input signals such that information about future inputs is encoded. Slow feature analysis (SFA) is a method for extracting slowly varying components from quickly varying input signals, thereby learning temporally invariant features. Here, we use the information bottleneck method to state an information-theoretic objective function for temporally local predictive coding. We then show that the linear case of SFA can be interpreted as a variant of predictive coding that maximizes the mutual information between the current output of the system and the input signal in the next time step. This demonstrates that the slowness principle and predictive coding are intimately related.

Communication system model for information rate evaluation of differential detection over time-varying channels

A communication system model for mutual information performance analysis of multiple-symbol differential M-phase shift keying over time-correlated, time-varying flat-fading communication channels is developed. This model is a finite-state Markov (FSM) equivalent channel representing the cascade of the differential encoder, FSM channel model and differential decoder. A state-space approach is used to model channel phase time correlations. The equivalent model falls in a class that facilitates the use of the forward backward algorithm, enabling the important information theoretic results to be evaluated. Using such a model, one is able to calculate mutual information for differential detection over time-varying fading channels with an essentially finite time set of correlations, including the Clarke fading channel. Using the equivalent channel, it is proved and corroborated by simulations that multiple-symbol differential detection preserves the channel information capacity when the observation interval approaches infinity.

Information theoretic measures of UHG graphs with low computational complexity

We introduce a novel graph class we call universal hierarchical graphs (UHG) whose topology can be found numerously in problems representing, e.g., temporal, spacial or general process structures of systems. For this graph class we show, that we can naturally assign two probability distributions, for nodes and for edges, which lead us directly to the definition of the entropy and joint entropy and, hence, mutual information establishing an information theory for this graph class. Furthermore, we provide some results under which conditions these constraint probability distributions maximize the corresponding entropy. Also, we demonstrate that these entropic measures can be computed efficiently which is a prerequisite for every large scale practical application and show some numerical examples. (c) 2007 Elsevier Inc. All rights reserved.

Generating a Word-Emotion Lexicon from #Emotional Tweets

Research in emotion analysis of text suggest that emotion lexicon based features are superior to corpus based n-gram features. However the static nature of the general purpose emotion lexicons make them less suited to social media analysis, where the need to adopt to changes in vocabulary usage and context is crucial. In this paper we propose a set of methods to extract a word-emotion lexicon automatically from an emotion labelled corpus of tweets. Our results confirm that the features derived from these lexicons outperform the standard Bag-of-words features when applied to an emotion classification task. Furthermore, a comparative analysis with both manually crafted lexicons and a state-of-the-art lexicon generated using Point-Wise Mutual Information, show that the lexicons generated from the proposed methods lead to significantly better classi- fication performance.

Image Segmentation Using Information Bottleneck Method

In image processing, segmentation algorithms constitute one of the main focuses of research. In this paper, new image segmentation algorithms based on a hard version of the information bottleneck method are presented. The objective of this method is to extract a compact representation of a variable, considered the input, with minimal loss of mutual information with respect to another variable, considered the output. First, we introduce a split-and-merge algorithm based on the definition of an information channel between a set of regions (input) of the image and the intensity histogram bins (output). From this channel, the maximization of the mutual information gain is used to optimize the image partitioning. Then, the merging process of the regions obtained in the previous phase is carried out by minimizing the loss of mutual information. From the inversion of the above channel, we also present a new histogram clustering algorithm based on the minimization of the mutual information loss, where now the input variable represents the histogram bins and the output is given by the set of regions obtained from the above split-and-merge algorithm. Finally, we introduce two new clustering algorithms which show how the information bottleneck method can be applied to the registration channel obtained when two multimodal images are correctly aligned. Different experiments on 2-D and 3-D images show the behavior of the proposed algorithms

View-dependent information theory quality measures for pixel sampling and scene discretization in flatland

In this paper, we present view-dependent information theory quality measures for pixel sampling and scene discretization in flatland. The measures are based on a definition for the mutual information of a line, and have a purely geometrical basis. Several algorithms exploiting them are presented and compare well with an existing one based on depth differences

An information theoretic framework for image segmentation

In this paper, an information theoretic framework for image segmentation is presented. This approach is based on the information channel that goes from the image intensity histogram to the regions of the partitioned image. It allows us to define a new family of segmentation methods which maximize the mutual information of the channel. Firstly, a greedy top-down algorithm which partitions an image into homogeneous regions is introduced. Secondly, a histogram quantization algorithm which clusters color bins in a greedy bottom-up way is defined. Finally, the resulting regions in the partitioning algorithm can optionally be merged using the quantized histogram

Single Synapse Information Coding in Intraburst Spike Patterns of Central Pattern Generator Motor Neurons

Burst firing is ubiquitous in nervous systems and has been intensively studied in central pattern generators (CPGs). Previous works have described subtle intraburst spike patterns (IBSPs) that, despite being traditionally neglected for their lack of relation to CPG motor function, were shown to be cell-type specific and sensitive to CPG connectivity. Here we address this matter by investigating how a bursting motor neuron expresses information about other neurons in the network. We performed experiments on the crustacean stomatogastric pyloric CPG, both in control conditions and interacting in real-time with computer model neurons. The sensitivity of postsynaptic to presynaptic IBSPs was inferred by computing their average mutual information along each neuron burst. We found that details of input patterns are nonlinearly and inhomogeneously coded through a single synapse into the fine IBSPs structure of the postsynaptic neuron following burst. In this way, motor neurons are able to use different time scales to convey two types of information simultaneously: muscle contraction (related to bursting rhythm) and the behavior of other CPG neurons (at a much shorter timescale by using IBSPs as information carriers). Moreover, the analysis revealed that the coding mechanism described takes part in a previously unsuspected information pathway from a CPG motor neuron to a nerve that projects to sensory brain areas, thus providing evidence of the general physiological role of information coding through IBSPs in the regulation of neuronal firing patterns in remote circuits by the CNS.

An information-theoretic approach to statistical dependence: Copula information

We discuss the connection between information and copula theories by showing that a copula can be employed to decompose the information content of a multivariate distribution into marginal and dependence components, with the latter quantified by the mutual information. We define the information excess as a measure of deviation from a maximum-entropy distribution. The idea of marginal invariant dependence measures is also discussed and used to show that empirical linear correlation underestimates the amplitude of the actual correlation in the case of non-Gaussian marginals. The mutual information is shown to provide an upper bound for the asymptotic empirical log-likelihood of a copula. An analytical expression for the information excess of T-copulas is provided, allowing for simple model identification within this family. We illustrate the framework in a financial data set. Copyright (C) EPLA, 2009