Modeling High-Dimensional Audio Sequences with Recurrent Neural Networks

Cette thèse étudie des modèles de séquences de haute dimension basés sur des réseaux de neurones récurrents (RNN) et leur application à la musique et à la parole. Bien qu'en principe les RNN puissent représenter les dépendances à long terme et la dynamique temporelle complexe propres aux séquences d'intérêt comme la vidéo, l'audio et la langue naturelle, ceux-ci n'ont pas été utilisés à leur plein potentiel depuis leur introduction par Rumelhart et al. (1986a) en raison de la difficulté de les entraîner efficacement par descente de gradient. Récemment, l'application fructueuse de l'optimisation Hessian-free et d'autres techniques d'entraînement avancées ont entraîné la recrudescence de leur utilisation dans plusieurs systèmes de l'état de l'art. Le travail de cette thèse prend part à ce développement. L'idée centrale consiste à exploiter la flexibilité des RNN pour apprendre une description probabiliste de séquences de symboles, c'est-à-dire une information de haut niveau associée aux signaux observés, qui en retour pourra servir d'à priori pour améliorer la précision de la recherche d'information. Par exemple, en modélisant l'évolution de groupes de notes dans la musique polyphonique, d'accords dans une progression harmonique, de phonèmes dans un énoncé oral ou encore de sources individuelles dans un mélange audio, nous pouvons améliorer significativement les méthodes de transcription polyphonique, de reconnaissance d'accords, de reconnaissance de la parole et de séparation de sources audio respectivement. L'application pratique de nos modèles à ces tâches est détaillée dans les quatre derniers articles présentés dans cette thèse. Dans le premier article, nous remplaçons la couche de sortie d'un RNN par des machines de Boltzmann restreintes conditionnelles pour décrire des distributions de sortie multimodales beaucoup plus riches. Dans le deuxième article, nous évaluons et proposons des méthodes avancées pour entraîner les RNN. Dans les quatre derniers articles, nous examinons différentes façons de combiner nos modèles symboliques à des réseaux profonds et à la factorisation matricielle non-négative, notamment par des produits d'experts, des architectures entrée/sortie et des cadres génératifs généralisant les modèles de Markov cachés. Nous proposons et analysons également des méthodes d'inférence efficaces pour ces modèles, telles la recherche vorace chronologique, la recherche en faisceau à haute dimension, la recherche en faisceau élagué et la descente de gradient. Finalement, nous abordons les questions de l'étiquette biaisée, du maître imposant, du lissage temporel, de la régularisation et du pré-entraînement.

A semi-blind reception scheme based on ICA for multiuser MIMO-STBC system

When the orthogonal space-time block code (STBC), or the Alamouti code, is applied on a multiple-input multiple-output (MIMO) communications system, the optimum reception can be achieved by a simple signal decoupling at the receiver. The performance, however, deteriorates significantly in presence of co-channel interference (CCI) from other users. In this paper, such CCI problem is overcome by applying the independent component analysis (ICA), a blind source separation algorithm. This is based on the fact that, if the transmission data from every transmit antenna are mutually independent, they can be effectively separated at the receiver with the principle of the blind source separation. Then equivalently, the CCI is suppressed. Although they are not required by the ICA algorithm itself, a small number of training data are necessary to eliminate the phase and order ambiguities at the ICA outputs, leading to a semi-blind approach. Numerical simulation is also shown to verify the proposed ICA approach in the multiuser MIMO system.

Automatic artefact removal from event-related potentials via clustering

This paper outlines a method for automatic artefact removal from multichannel recordings of event-related potentials (ERPs). The proposed method is based on, firstly, separation of the ERP recordings into independent components using the method of temporal decorrelation source separation (TDSEP). Secondly, the novel lagged auto-mutual information clustering (LAMIC) algorithm is used to cluster the estimated components, together with ocular reference signals, into clusters corresponding to cerebral and non-cerebral activity. Thirdly, the components in the cluster which contains the ocular reference signals are discarded. The remaining components are then recombined to reconstruct the clean ERPs.

Automated artifact removal from the electroencephalogram; a comparative study

Contamination of the electroencephalogram (EEG) by artifacts greatly reduces the quality of the recorded signals. There is a need for automated artifact removal methods. However, such methods are rarely evaluated against one another via rigorous criteria, with results often presented based upon visual inspection alone. This work presents a comparative study of automatic methods for removing blink, electrocardiographic, and electromyographic artifacts from the EEG. Three methods are considered; wavelet, blind source separation (BSS), and multivariate singular spectrum analysis (MSSA)-based correction. These are applied to data sets containing mixtures of artifacts. Metrics are devised to measure the performance of each method. The BSS method is seen to be the best approach for artifacts of high signal to noise ratio (SNR). By contrast, MSSA performs well at low SNRs but at the expense of a large number of false positive corrections.

Hushållsavfall i Borlänge kommun : analyser av material en och flerfamiljshus

In Borlänge, source separation has been the basis for management of household waste for over five years. This report reviews today?s system and gives a model for further follow-up through waste grouping. In the basic system waste is separated into three fractions: biodegradable, waste to energy and waste to landfill. All waste is packed in plastic bags, put in separate containers for each fraction, and collected from the property. Separate analyses were made of waste from single family houses and apartment buildings. The amount of waste per household and week, number of non-sorted bags, purity, recovery rate and density of each fraction was calculated. The amount of packaging collected together with the household waste is given. Material collected under the Swedish law of Producers? Responsibility is not covered in this report.

A novel monotonic fixed-point algorithm for l1-regularized least square vector and matrix problem

Least square problem with l₁ regularization has been proposed as a promising method for sparse signal reconstruction (e.g., basis pursuit de-noising and compressed sensing) and feature selection (e.g., the Lasso algorithm) in signal processing, statistics, and related fields. These problems can be cast as l₁-regularized least-square program (LSP). In this paper, we propose a novel monotonic fixed point method to solve large-scale l₁-regularized LSP. And we also prove the stability and convergence of the proposed method. Furthermore we generalize this method to least square matrix problem and apply it in nonnegative matrix factorization (NMF). The method is illustrated on sparse signal reconstruction, partner recognition and blind source separation problems, and the method tends to convergent faster and sparser than other l₁-regularized algorithms.

Sparse Representations of Image via Overcomplete Dictionary Learned by Adaptive Non-orthogonal Sparsifying Transform

How to learn an over complete dictionary for sparse representations of image is an important topic in machine learning, sparse coding, blind source separation, etc. The so-called K-singular value decomposition (K-SVD) method [3] is powerful for this purpose, however, it is too time-consuming to apply. Recently, an adaptive orthogonal sparsifying transform (AOST) method has been developed to learn the dictionary that is faster. However, the corresponding coefficient matrix may not be as sparse as that of K-SVD. For solving this problem, in this paper, a non-orthogonal iterative match method is proposed to learn the dictionary. By using the approach of sequentially extracting columns of the stacked image blocks, the non-orthogonal atoms of the dictionary are learned adaptively, and the resultant coefficient matrix is sparser. Experiment results show that the proposed method can yield effective dictionaries and the resulting image representation is sparser than AOST.

Nonorthogonal approximate joint diagonalization with well-conditioned diagonalizers

To make the results reasonable, existing joint diagonalization algorithms have imposed a variety of constraints on diagonalizers. Actually, those constraints can be imposed uniformly by minimizing the condition number of diagonalizers. Motivated by this, the approximate joint diagonalization problem is reviewed as a multiobjective optimization problem for the first time. Based on this, a new algorithm for nonorthogonal joint diagonalization is developed. The new algorithm yields diagonalizers which not only minimize the diagonalization error but also have as small condition numbers as possible. Meanwhile, degenerate solutions are avoided strictly. Besides, the new algorithm imposes few restrictions on the target set of matrices to be diagonalized, which makes it widely applicable. Primary results on convergence are presented and we also show that, for exactly jointly diagonalizable sets, no local minima exist and the solutions are unique under mild conditions. Extensive numerical simulations illustrate the performance of the algorithm and provide comparison with other leading diagonalization methods. The practical use of our algorithm is shown for blind source separation (BSS) problems, especially when ill-conditioned mixing matrices are involved.

Nonnegative Matrix Factorization Applied to Nonlinear Speech and Image Cryptosystems

Nonnegative matrix factorization (NMF) is widely used in signal separation and image compression. Motivated by its successful applications, we propose a new cryptosystem based on NMF, where the nonlinear mixing (NLM) model with a strong noise is introduced for encryption and NMF is used for decryption. The security of the cryptosystem relies on following two facts: 1) the constructed multivariable nonlinear function is not invertible; 2) the process of NMF is unilateral, if the inverse matrix of the constructed linear mixing matrix is not nonnegative. Comparing with Lin's method (2006) that is a theoretical scheme using one-time padding in the cryptosystem, our cipher can be used repeatedly for the practical request, i.e., multitme padding is used in our cryptosystem. Also, there is no restriction on statistical characteristics of the ciphers and the plaintexts. Thus, more signals can be processed (successfully encrypted and decrypted), no matter they are correlative, sparse, or Gaussian. Furthermore, instead of the number of zero-crossing-based method that is often unstable in encryption and decryption, an improved method based on the kurtosis of the signals is introduced to solve permutation ambiguities in waveform reconstruction. Simulations are given to illustrate security and availability of our cryptosystem.

Mixing Matrix Estimation From Sparse Mixtures With Unknown Number of Sources

In blind source separation, many methods have been proposed to estimate the mixing matrix by exploiting sparsity. However, they often need to know the source number a priori, which is very inconvenient in practice. In this paper, a new method, namely nonlinear projection and column masking (NPCM), is proposed to estimate the mixing matrix. A major advantage of NPCM is that it does not need any knowledge of the source number. In NPCM, the objective function is based on a nonlinear projection and its maxima just correspond to the columns of the mixing matrix. Thus a column can be estimated first by locating a maximum and then deflated by a masking operation. This procedure is repeated until the evaluation of the objective function decreases to zero dramatically. Thus the mixing matrix and the number of sources are estimated simultaneously. Because the masking procedure may result in some small and useless local maxima, particle swarm optimization (PSO) is introduced to optimize the objective function. Feasibility and efficiency of PSO are also discussed. Comparative experimental results show the efficiency of NPCM, especially in the cases where the number of sources is unknown and the sources are relatively less sparse.

Blind extraction of cyclostationary signal from convolutional mixtures

Extracting a signal of interest from available measurements is a challenging problem. One property which can be utilized to extract the signal is cyclostationarity, which exists in many signals. Various blind source separation methods based on cyclostationarity have been reported in the literature but they assume that the mixing system is instantaneous. In this paper, we propose a method for blind extraction of cyclostationary signal from convolutional mixtures. Given that the signal of interest has a unique cyclostationary frequency and the sensors are placed close to the concerned signal, we show that the signal of interest can be estimated from the measured data. Simulations results show the effectiveness of our method.

Blind estimation of MIMO relay channels

In this paper, we integrate two blind source separation (BSS) methods to estimate the individual channel state information (CSI) for the source-relay and relay-destination links of three-node two-hop multiple-input multiple-output (MIMO) relay systems. In particular, we propose a first-order Z-domain precoding technique for the blind estimation of the relay-destination channel matrix, while an algorithm based on the constant modulus and mutual information properties is developed to estimate the source-relay channel matrix. Compared with training-based MIMO relay channel estimation approaches, our algorithm has a better bandwidth efficiency as no bandwidth is wasted for sending the training sequences. Numerical examples are shown to demonstrate the performance of the proposed algorithm. © 2014 IEEE.

Separating spatially correlated signals using first-order precoders

This paper deals with blind separation of spatially correlated signals mixed by an instantaneous system. Taking advantage of the fact that the source signals are accessible in some man-made systems such as wireless communication systems, we preprocess the source signals in transmitters by a set of properly designed first-order precoders and then the coded signals are transmitted. At the receiving side, information about the precoders are utilized to perform signal separation. Compared with the existing precoder-based methods, the new method only employs the simplest first-order precoders, which reduces the delay in data transmission and is easier to implement in practical applications.