Information geometric measurements of generalisation

Neural networks can be regarded as statistical models, and can be analysed in a Bayesian framework. Generalisation is measured by the performance on independent test data drawn from the same distribution as the training data. Such performance can be quantified by the posterior average of the information divergence between the true and the model distributions. Averaging over the Bayesian posterior guarantees internal coherence; Using information divergence guarantees invariance with respect to representation. The theory generalises the least mean squares theory for linear Gaussian models to general problems of statistical estimation. The main results are: (1)~the ideal optimal estimate is always given by average over the posterior; (2)~the optimal estimate within a computational model is given by the projection of the ideal estimate to the model. This incidentally shows some currently popular methods dealing with hyperpriors are in general unnecessary and misleading. The extension of information divergence to positive normalisable measures reveals a remarkable relation between the dlt dual affine geometry of statistical manifolds and the geometry of the dual pair of Banach spaces Ld and Ldd. It therefore offers conceptual simplification to information geometry. The general conclusion on the issue of evaluating neural network learning rules and other statistical inference methods is that such evaluations are only meaningful under three assumptions: The prior P(p), describing the environment of all the problems; the divergence Dd, specifying the requirement of the task; and the model Q, specifying available computing resources.

Advanced mean field methods:theory and practice

A major problem in modern probabilistic modeling is the huge computational complexity involved in typical calculations with multivariate probability distributions when the number of random variables is large. Because exact computations are infeasible in such cases and Monte Carlo sampling techniques may reach their limits, there is a need for methods that allow for efficient approximate computations. One of the simplest approximations is based on the mean field method, which has a long history in statistical physics. The method is widely used, particularly in the growing field of graphical models. Researchers from disciplines such as statistical physics, computer science, and mathematical statistics are studying ways to improve this and related methods and are exploring novel application areas. Leading approaches include the variational approach, which goes beyond factorizable distributions to achieve systematic improvements; the TAP (Thouless-Anderson-Palmer) approach, which incorporates correlations by including effective reaction terms in the mean field theory; and the more general methods of graphical models. Bringing together ideas and techniques from these diverse disciplines, this book covers the theoretical foundations of advanced mean field methods, explores the relation between the different approaches, examines the quality of the approximation obtained, and demonstrates their application to various areas of probabilistic modeling.

Mean field methods for classification with Gaussian processes

We discuss the Application of TAP mean field methods known from Statistical Mechanics of disordered systems to Bayesian classification with Gaussian processes. In contrast to previous applications, no knowledge about the distribution of inputs is needed. Simulation results for the Sonar data set are given.

Mean field methods for Gaussian process classifiers

DUE TO COPYRIGHT RESTRICTIONS ONLY AVAILABLE FOR CONSULTATION AT ASTON UNIVERSITY LIBRARY AND INFORMATION SERVICES WITH PRIOR ARRANGEMENT

Stochastic stability analysis for the constant-modulus algorithm

We derive an easy-to-compute approximate bound for the range of step-sizes for which the constant-modulus algorithm (CMA) will remain stable if initialized close to a minimum of the CM cost function. Our model highlights the influence, of the signal constellation used in the transmission system: for smaller variation in the modulus of the transmitted symbols, the algorithm will be more robust, and the steady-state misadjustment will be smaller. The theoretical results are validated through several simulations, for long and short filters and channels.

Comunicações digitais através de ultra-sons em ambientes subaquáticos

Actualmente, as comunicações digitais em ambientes subaquáticos representam uma necessidade humana e um desafio à engenharia de sistemas. A maioria das aplicações dirigidas a meios subaquáticos recorre a sinais acústicos, já que relativamente a sinais de natureza electromagnética ou óptica, os sinais acústicos desfrutam de uma atenuação reduzida. Contudo, os meios subaquáticos possuem um comportamento complexo relativamente à propagação de sinais acústico, sendo caracterizados como canais dispersivos, no domínio do tempo e no domínio da frequência. Como os sistemas acústicos possuem uma largura de banda limitada, requer-se que um sistema de comunicação acústico subaquático seja, simultaneamente, eficiente na gestão dos recursos que possui e eficaz nos mecanismos que implementa para ultrapassar as limitações providenciadas pelo meio. Enquadrada neste âmbito, a presente dissertação propõe uma solução completa de um sistema de comunicação digital. Apresenta-se o dimensionamento de uma plataforma genérica, de baixo custo, que suporta a transmissão e recepção de sinais acústicos num ambiente subaquático. Através desta, verifica-se num meio subaquático real se as características deste ambiente coincidem com a informação presente na respectiva literatura científica. Adicionalmente, são analisadas quais as técnicas de processamento de sinal mais adequadas às comunicações digitais neste meio. Converge-se para uma solução baseada na modulação OFDM (Orthogonal Frequency Division Multiplexing), que alcança uma eficiência espectral de 2,95 bit/s/Hz, a um ritmo binário máximo de 103,288 kbit/s, numa largura de banda de 35 kHz. Através desta técnica, obtém-se elevada robustez aos efeitos de dispersão temporal do canal de comunicação, minimizando de forma eficaz a distorção de sinal devido à interferência inter-simbólica. Esta modulação multi-portadora recorre a sub-portadoras ortogonais, maximizando desta forma a eficiência espectral do sistema. Contudo, a ortogonalidade destas componentes pode ser comprometida pelo espalhamento de Doppler introduzido pelo canal subaquático. Para minimizar este efeito, é implementado um algoritmo adaptativo baseado na técnica LMS (Least Mean Squares).

Factorization of elliptic boundary value problems by invariant embedding and application to overdetermined problems

Dissertação para obtenção do Grau de Doutor em Matemática

An LMS style variable tap-length algorithm for structure adaptation

Searching for the optimum tap-length that best balances the complexity and steady-state performance of an adaptive filter has attracted attention recently. Among existing algorithms that can be found in the literature, two of which, namely the segmented filter (SF) and gradient descent (GD) algorithms, are of particular interest as they can search for the optimum tap-length quickly. In this paper, at first, we carefully compare the SF and GD algorithms and show that the two algorithms are equivalent in performance under some constraints, but each has advantages/disadvantages relative to the other. Then, we propose an improved variable tap-length algorithm using the concept of the pseudo fractional tap-length (FT). Updating the tap-length with instantaneous errors in a style similar to that used in the stochastic gradient [or least mean squares (LMS)] algorithm, the proposed FT algorithm not only retains the advantages from both the SF and the GD algorithms but also has significantly less complexity than existing algorithms. Both performance analysis and numerical simulations are given to verify the new proposed algorithm.

Decision-directed feedback equalisation for QAM signals

Proposed is a symbol-based decision-directed algorithm for blind equalisation of quadrature amplitude modulation (QAM) signals using a decision feedback scheme. Independently of QAM order, it presents: (i) an error equal to zero when the equaliser output coincides with the transmitted signal; (ii) simultaneous recovery of the modulus and phase of the signal; (iii) a misadjustment close to that of the normalised least-mean squares algorithm; (iv) fast convergence; and (v) the avoidance of degenerative solutions. Additionally, its stability is ensured when the step-size is properly chosen.

Dynamic chanel model LMS updating for RSS-based localization

Received signal strength-based localization systems usually rely on a calibration process that aims at characterizing the propagation channel. However, due to the changing environmental dynamics, the behavior of the channel may change after some time, thus, recalibration processes are necessary to maintain the positioning accuracy. This paper proposes a dynamic calibration method to initially calibrate and subsequently update the parameters of the propagation channel model using a Least Mean Squares approach. The method assumes that each anchor node in the localization infrastructure is characterized by its own propagation channel model. In practice, a set of sniffers is used to collect RSS samples, which will be used to automatically calibrate each channel model by iteratively minimizing the positioning error. The proposed method is validated through numerical simulation, showing that the positioning error of the mobile nodes is effectively reduced. Furthermore, the method has a very low computational cost; therefore it can be used in real-time operation for wireless resource-constrained nodes.

Cancelación activa del ruido utilizando el KIT TMS320C5515 EZDSP

El control, o cancelación activa de ruido, consiste en la atenuación del ruido presente en un entorno acústico mediante la emisión de una señal igual y en oposición de fase al ruido que se desea atenuar. La suma de ambas señales en el medio acústico produce una cancelación mutua, de forma que el nivel de ruido resultante es mucho menor al inicial. El funcionamiento de estos sistemas se basa en los principios de comportamiento de los fenómenos ondulatorios descubiertos por Augustin-Jean Fresnel, Christiaan Huygens y Thomas Young entre otros. Desde la década de 1930, se han desarrollado prototipos de sistemas de control activo de ruido, aunque estas primeras ideas eran irrealizables en la práctica o requerían de ajustes manuales cada poco tiempo que hacían inviable su uso. En la década de 1970, el investigador estadounidense Bernard Widrow desarrolla la teoría de procesado adaptativo de señales y el algoritmo de mínimos cuadrados LMS. De este modo, es posible implementar filtros digitales cuya respuesta se adapte de forma dinámica a las condiciones variables del entorno. Con la aparición de los procesadores digitales de señal en la década de 1980 y su evolución posterior, se abre la puerta para el desarrollo de sistemas de cancelación activa de ruido basados en procesado de señal digital adaptativo. Hoy en día, existen sistemas de control activo de ruido implementados en automóviles, aviones, auriculares o racks de equipamiento profesional. El control activo de ruido se basa en el algoritmo fxlms, una versión modificada del algoritmo LMS de filtrado adaptativo que permite compensar la respuesta acústica del entorno. De este modo, se puede filtrar una señal de referencia de ruido de forma dinámica para emitir la señal adecuada que produzca la cancelación. Como el espacio de cancelación acústica está limitado a unas dimensiones de la décima parte de la longitud de onda, sólo es viable la reducción de ruido en baja frecuencia. Generalmente se acepta que el límite está en torno a 500 Hz. En frecuencias medias y altas deben emplearse métodos pasivos de acondicionamiento y aislamiento, que ofrecen muy buenos resultados. Este proyecto tiene como objetivo el desarrollo de un sistema de cancelación activa de ruidos de carácter periódico, empleando para ello electrónica de consumo y un kit de desarrollo DSP basado en un procesador de muy bajo coste. Se han desarrollado una serie de módulos de código para el DSP escritos en lenguaje C, que realizan el procesado de señal adecuado a la referencia de ruido. Esta señal procesada, una vez emitida, produce la cancelación acústica. Empleando el código implementado, se han realizado pruebas que generan la señal de ruido que se desea eliminar dentro del propio DSP. Esta señal se emite mediante un altavoz que simula la fuente de ruido a cancelar, y mediante otro altavoz se emite una versión filtrada de la misma empleando el algoritmo fxlms. Se han realizado pruebas con distintas versiones del algoritmo, y se han obtenido atenuaciones de entre 20 y 35 dB medidas en márgenes de frecuencia estrechos alrededor de la frecuencia del generador, y de entre 8 y 15 dB medidas en banda ancha. ABSTRACT. Active noise control consists on attenuating the noise in an acoustic environment by emitting a signal equal but phase opposed to the undesired noise. The sum of both signals results in mutual cancellation, so that the residual noise is much lower than the original. The operation of these systems is based on the behavior principles of wave phenomena discovered by Augustin-Jean Fresnel, Christiaan Huygens and Thomas Young. Since the 1930’s, active noise control system prototypes have been developed, though these first ideas were practically unrealizable or required manual adjustments very often, therefore they were unusable. In the 1970’s, American researcher Bernard Widrow develops the adaptive signal processing theory and the Least Mean Squares algorithm (LMS). Thereby, implementing digital filters whose response adapts dynamically to the variable environment conditions, becomes possible. With the emergence of digital signal processors in the 1980’s and their later evolution, active noise cancellation systems based on adaptive signal processing are attained. Nowadays active noise control systems have been successfully implemented on automobiles, planes, headphones or racks for professional equipment. Active noise control is based on the fxlms algorithm, which is actually a modified version of the LMS adaptive filtering algorithm that allows compensation for the acoustic response of the environment. Therefore it is possible to dynamically filter a noise reference signal to obtain the appropriate cancelling signal. As the noise cancellation space is limited to approximately one tenth of the wavelength, noise attenuation is only viable for low frequencies. It is commonly accepted the limit of 500 Hz. For mid and high frequencies, conditioning and isolating passive techniques must be used, as they produce very good results. The objective of this project is to develop a noise cancellation system for periodic noise, by using consumer electronics and a DSP development kit based on a very-low-cost processor. Several C coded modules have been developed for the DSP, implementing the appropriate signal processing to the noise reference. This processed signal, once emitted, results in noise cancellation. The developed code has been tested by generating the undesired noise signal in the DSP. This signal is emitted through a speaker simulating the noise source to be removed, and another speaker emits an fxlms filtered version of the same signal. Several versions of the algorithm have been tested, obtaining attenuation levels around 20 – 35 dB measured in a tight bandwidth around the generator frequency, or around 8 – 15 dB measured in broadband.

Learning Preferences with Kernel-Based Methods

Learning of preference relations has recently received significant attention in machine learning community. It is closely related to the classification and regression analysis and can be reduced to these tasks. However, preference learning involves prediction of ordering of the data points rather than prediction of a single numerical value as in case of regression or a class label as in case of classification. Therefore, studying preference relations within a separate framework facilitates not only better theoretical understanding of the problem, but also motivates development of the efficient algorithms for the task. Preference learning has many applications in domains such as information retrieval, bioinformatics, natural language processing, etc. For example, algorithms that learn to rank are frequently used in search engines for ordering documents retrieved by the query. Preference learning methods have been also applied to collaborative filtering problems for predicting individual customer choices from the vast amount of user generated feedback. In this thesis we propose several algorithms for learning preference relations. These algorithms stem from well founded and robust class of regularized least-squares methods and have many attractive computational properties. In order to improve the performance of our methods, we introduce several non-linear kernel functions. Thus, contribution of this thesis is twofold: kernel functions for structured data that are used to take advantage of various non-vectorial data representations and the preference learning algorithms that are suitable for different tasks, namely efficient learning of preference relations, learning with large amount of training data, and semi-supervised preference learning. Proposed kernel-based algorithms and kernels are applied to the parse ranking task in natural language processing, document ranking in information retrieval, and remote homology detection in bioinformatics domain. Training of kernel-based ranking algorithms can be infeasible when the size of the training set is large. This problem is addressed by proposing a preference learning algorithm whose computation complexity scales linearly with the number of training data points. We also introduce sparse approximation of the algorithm that can be efficiently trained with large amount of data. For situations when small amount of labeled data but a large amount of unlabeled data is available, we propose a co-regularized preference learning algorithm. To conclude, the methods presented in this thesis address not only the problem of the efficient training of the algorithms but also fast regularization parameter selection, multiple output prediction, and cross-validation. Furthermore, proposed algorithms lead to notably better performance in many preference learning tasks considered.

Analysis of carrier phase extraction methods in 112-Gbit/s NRZ-PDM-QPSK coherent transmission system

We present a comparative analysis on three carrier phase extraction approaches, including a one-tap normalized least mean square method, a block-average method, and a Viterbi-Viterbi method, in coherent transmission system considering equalization enhanced phase noise. © OSA 2012.

Páros összehasonlításon alapuló pontozási eljárások monotonitása: önkonzisztencia és önkonzisztens monotonitás. Adalékok a pontozási eljárások axiomatikus tárgyalásához = Monotonicity of paired comparison-based ranking methods: self-consistency and self-consistent monotonicity. Contributions to an axiomatic analysis of scoring methods

A cikk a páros összehasonlításokon alapuló pontozási eljárásokat tárgyalja axiomatikus megközelítésben. A szakirodalomban számos értékelő függvényt javasoltak erre a célra, néhány karakterizációs eredmény is ismert. Ennek ellenére a megfelelő módszer kiválasztása nem egy-szerű feladat, a különböző tulajdonságok bevezetése elsősorban ebben nyújthat segítséget. Itt az összehasonlított objektumok teljesítményén érvényesülő monotonitást tárgyaljuk az önkonzisztencia és önkonzisztens monotonitás axiómákból kiindulva. Bemutatásra kerülnek lehetséges gyengítéseik és kiterjesztéseik, illetve egy, az irreleváns összehasonlításoktól való függetlenséggel kapcsolatos lehetetlenségi tétel is. A tulajdonságok teljesülését három eljárásra, a klasszikus pontszám eljárásra, az ezt továbbfejlesztő általánosított sorösszegre és a legkisebb négyzetek módszerére vizsgáljuk meg, melyek mindegyike egy lineáris egyenletrendszer megoldásaként számítható. A kapott eredmények új szempontokkal gazdagítják a pontozási eljárás megválasztásának kérdését. _____ The paper provides an axiomatic analysis of some scoring procedures based on paired comparisons. Several methods have been proposed for these generalized tournaments, some of them have been also characterized by a set of properties. The choice of an appropriate method is supported by a discussion of their theoretical properties. In the paper we focus on the connections of self-consistency and self-consistent-monotonicity, two axioms based on the comparisons of object's performance. The contradiction of self-consistency and independence of irrel-evant matches is revealed, as well as some possible reductions and extensions of these properties. Their satisfiability is examined through three scoring procedures, the score, generalised row sum and least squares methods, each of them is calculated as a solution of a system of linear equations. Our results contribute to the problem of finding a proper paired comparison based scoring method.

Distributed Estimation Over an Adaptive Incremental Network Based on the Affine Projection Algorithm

We study the problem of distributed estimation based on the affine projection algorithm (APA), which is developed from Newton`s method for minimizing a cost function. The proposed solution is formulated to ameliorate the limited convergence properties of least-mean-square (LMS) type distributed adaptive filters with colored inputs. The analysis of transient and steady-state performances at each individual node within the network is developed by using a weighted spatial-temporal energy conservation relation and confirmed by computer simulations. The simulation results also verify that the proposed algorithm provides not only a faster convergence rate but also an improved steady-state performance as compared to an LMS-based scheme. In addition, the new approach attains an acceptable misadjustment performance with lower computational and memory cost, provided the number of regressor vectors and filter length parameters are appropriately chosen, as compared to a distributed recursive-least-squares (RLS) based method.