Learning the Structure of Deep Sparse Graphical Models

Deep belief networks are a powerful way to model complex probability distributions. However, learning the structure of a belief network, particularly one with hidden units, is difficult. The Indian buffet process has been used as a nonparametric Bayesian prior on the directed structure of a belief network with a single infinitely wide hidden layer. In this paper, we introduce the cascading Indian buffet process (CIBP), which provides a nonparametric prior on the structure of a layered, directed belief network that is unbounded in both depth and width, yet allows tractable inference. We use the CIBP prior with the nonlinear Gaussian belief network so each unit can additionally vary its behavior between discrete and continuous representations. We provide Markov chain Monte Carlo algorithms for inference in these belief networks and explore the structures learned on several image data sets.

Nonparametric Bayesian sparse factor models with application to gene expression modeling

A nonparametric Bayesian extension of Factor Analysis (FA) is proposed where observed data $\mathbf{Y}$ is modeled as a linear superposition, $\mathbf{G}$, of a potentially infinite number of hidden factors, $\mathbf{X}$. The Indian Buffet Process (IBP) is used as a prior on $\mathbf{G}$ to incorporate sparsity and to allow the number of latent features to be inferred. The model's utility for modeling gene expression data is investigated using randomly generated data sets based on a known sparse connectivity matrix for E. Coli, and on three biological data sets of increasing complexity.

Bayesian and L1 Approaches to Sparse Unsupervised Learning

The use of L1 regularisation for sparse learning has generated immense research interest, with successful application in such diverse areas as signal acquisition, image coding, genomics and collaborative filtering. While existing work highlights the many advantages of L1 methods, in this paper we find that L1 regularisation often dramatically underperforms in terms of predictive performance when compared with other methods for inferring sparsity. We focus on unsupervised latent variable models, and develop L1 minimising factor models, Bayesian variants of "L1", and Bayesian models with a stronger L0-like sparsity induced through spike-and-slab distributions. These spike-and-slab Bayesian factor models encourage sparsity while accounting for uncertainty in a principled manner and avoiding unnecessary shrinkage of non-zero values. We demonstrate on a number of data sets that in practice spike-and-slab Bayesian methods outperform L1 minimisation, even on a computational budget. We thus highlight the need to re-assess the wide use of L1 methods in sparsity-reliant applications, particularly when we care about generalising to previously unseen data, and provide an alternative that, over many varying conditions, provides improved generalisation performance.

On diagonal-Schur complements of block diagonally dominant matrices

It is known that the diagonal-Schur complements of strictly diagonally dominant matrices are strictly diagonally dominant matrices [J.Z. Liu, Y.Q. Huang, Some properties on Schur complements of H-matrices and diagonally dominant matrices, Linear Algebra Appl. 389 (2004) 365-380], and the same is true for nonsingular H-matrices [J.Z. Liu, J.C. Li, Z.T. Huang, X. Kong, Some properties of Schur complements and diagonal-Schur complements of diagonally dominant matrices, Linear Algebra Appl. 428 (2008) 1009-1030]. In this paper, we research the properties on diagonal-Schur complements of block diagonally dominant matrices and prove that the diagonal-Schur complements of block strictly diagonally dominant matrices are block strictly diagonally dominant matrices, and the same holds for generalized block strictly diagonally dominant matrices. (C) 2010 Elsevier Inc. All rights reserved.

Some new bounds on eigenvalues of the Hadamard product and the Fan product of matrices

Let A and B be nonsingular M-matrices. A lower bound on the minimum eigenvalue q(B circle A(-1)) for the Hadamard product of A(-1) and B, and a lower bound on the minimum eigenvalue q(A star B) for the Fan product of A and B are given. In addition, an upper bound on the spectral radius rho(A circle B) of nonnegative matrices A and B is also obtained. These bounds improve several existing results in some cases and the estimating formulas are easier to calculate for they are only depending on the entries of matrices A and B. (C) 2009 Elsevier Inc. All rights reserved.

Single base-resolution methylome of the silkworm reveals a sparse epigenomic map

Epigenetic regulation in insects may have effects on diverse biological processes. Here we survey the methylome of a model insect, the silkworm Bombyx mori, at single-base resolution using Illumina high-throughput bisulfite sequencing (MethylC-Seq). We conservatively estimate that 0.11% of genomic cytosines are methylcytosines, all of which probably occur in CG dinucleotides. CG methylation is substantially enriched in gene bodies and is positively correlated with gene expression levels, suggesting it has a positive role in gene transcription. We find that transposable elements, promoters and ribosomal DNAs are hypomethylated, but in contrast, genomic loci matching small RNAs in gene bodies are densely methylated. This work contributes to our understanding of epigenetics in insects, and in contrast to previous studies of the highly methylated genomes of Arabidopsis(1) and human(2), demonstrates a strategy for sequencing the epigenomes of organisms such as insects that have low levels of methylation.

Fast L0-based sparse signal recovery

This paper develops an algorithm for finding sparse signals from limited observations of a linear system. We assume an adaptive Gaussian model for sparse signals. This model results in a least square problem with an iteratively reweighted L2 penalty that approximates the L0-norm. We propose a fast algorithm to solve the problem within a continuation framework. In our examples, we show that the correct sparsity map and sparsity level are gradually learnt during the iterations even when the number of observations is reduced, or when observation noise is present. In addition, with the help of sophisticated interscale signal models, the algorithm is able to recover signals to a better accuracy and with reduced number of observations than typical L1-norm and reweighted L1 norm methods. ©2010 IEEE.