Regularization for sparsity in statistical analysis and machine learning

Pragmatism is the leading motivation of regularization. We can understand regularization as a modification of the maximum-likelihood estimator so that a reasonable answer could be given in an unstable or ill-posed situation. To mention some typical examples, this happens when fitting parametric or non-parametric models with more parameters than data or when estimating large covariance matrices. Regularization is usually used, in addition, to improve the bias-variance tradeoff of an estimation. Then, the definition of regularization is quite general, and, although the introduction of a penalty is probably the most popular type, it is just one out of multiple forms of regularization. In this dissertation, we focus on the applications of regularization for obtaining sparse or parsimonious representations, where only a subset of the inputs is used. A particular form of regularization, L1-regularization, plays a key role for reaching sparsity. Most of the contributions presented here revolve around L1-regularization, although other forms of regularization are explored (also pursuing sparsity in some sense). In addition to present a compact review of L1-regularization and its applications in statistical and machine learning, we devise methodology for regression, supervised classification and structure induction of graphical models. Within the regression paradigm, we focus on kernel smoothing learning, proposing techniques for kernel design that are suitable for high dimensional settings and sparse regression functions. We also present an application of regularized regression techniques for modeling the response of biological neurons. Supervised classification advances deal, on the one hand, with the application of regularization for obtaining a na¨ıve Bayes classifier and, on the other hand, with a novel algorithm for brain-computer interface design that uses group regularization in an efficient manner. Finally, we present a heuristic for inducing structures of Gaussian Bayesian networks using L1-regularization as a filter. El pragmatismo es la principal motivación de la regularización. Podemos entender la regularización como una modificación del estimador de máxima verosimilitud, de tal manera que se pueda dar una respuesta cuando la configuración del problema es inestable. A modo de ejemplo, podemos mencionar el ajuste de modelos paramétricos o no paramétricos cuando hay más parámetros que casos en el conjunto de datos, o la estimación de grandes matrices de covarianzas. Se suele recurrir a la regularización, además, para mejorar el compromiso sesgo-varianza en una estimación. Por tanto, la definición de regularización es muy general y, aunque la introducción de una función de penalización es probablemente el método más popular, éste es sólo uno de entre varias posibilidades. En esta tesis se ha trabajado en aplicaciones de regularización para obtener representaciones dispersas, donde sólo se usa un subconjunto de las entradas. En particular, la regularización L1 juega un papel clave en la búsqueda de dicha dispersión. La mayor parte de las contribuciones presentadas en la tesis giran alrededor de la regularización L1, aunque también se exploran otras formas de regularización (que igualmente persiguen un modelo disperso). Además de presentar una revisión de la regularización L1 y sus aplicaciones en estadística y aprendizaje de máquina, se ha desarrollado metodología para regresión, clasificación supervisada y aprendizaje de estructura en modelos gráficos. Dentro de la regresión, se ha trabajado principalmente en métodos de regresión local, proponiendo técnicas de diseño del kernel que sean adecuadas a configuraciones de alta dimensionalidad y funciones de regresión dispersas. También se presenta una aplicación de las técnicas de regresión regularizada para modelar la respuesta de neuronas reales. Los avances en clasificación supervisada tratan, por una parte, con el uso de regularización para obtener un clasificador naive Bayes y, por otra parte, con el desarrollo de un algoritmo que usa regularización por grupos de una manera eficiente y que se ha aplicado al diseño de interfaces cerebromáquina. Finalmente, se presenta una heurística para inducir la estructura de redes Bayesianas Gaussianas usando regularización L1 a modo de filtro.

Diui Thome aquinatis in tertio sententiarum libro i[n]terpretatio [et] expositio sum / a Thimotheo Veronense canonico regulari theologie scientissimo per[quam] diligenter visa recognita, erroribusque purgata [et] quantum anniti ars potuit fideliter impres

Pie de imp. tomado del colofón

Quinta pars huius operis co[n]tine[n]s postillam / d[omi]ni Hugonis Cardinalis super quattuor eua[n]gelia s[e]c[un]d[u]m Mattheu[m], Marcum, Lucam, Iohannem.

Sign.: a8, b6, c8, d6, e8, f6, g8, h6, i8, k6, l8, m6, n8, o6, p8, q6, r8, s6, t8, u6, x8, y6, z8, A6, B8, C6, D8, E6, F8, G6, H8, I6, K8, L6, M8, N6, O8, P6, Q8, R6, S8, T6, U8, X6, Y8, Z6, 2A8, 2B6, 2C8, 2D-2F6

Reglas de los cinco ordenes de Arquitectura de Vignola : con vn orden Dórico de Posidonia, y vn apéndice que contiene las lecciones elementales de las sombras en la Arquitectura, demostradas por principios naturales / por C.M. Delagardette ... ; dibuxado

Sign.: []1, *2, A-K2, L1

Mesh traversal and sorting for efficient memory usage in scientific codes

Applications that operate on meshes are very popular in High Performance Computing (HPC) environments. In the past, many techniques have been developed in order to optimize the memory accesses for these datasets. Different loop transformations and domain decompositions are com- monly used for structured meshes. However, unstructured grids are more challenging. The memory accesses, based on the mesh connectivity, do not map well to the usual lin- ear memory model. This work presents a method to improve the memory performance which is suitable for HPC codes that operate on meshes. We develop a method to adjust the sequence in which the data are used inside the algorithm, by means of traversing and sorting the mesh. This sorted mesh can be transferred sequentially to the lower memory levels and allows for minimum data transfer requirements. The method also reduces the lower memory requirements dra- matically: up to 63% of the L1 cache misses are removed in a traditional cache system. We have obtained speedups of up to 2.58 on memory operations as measured in a general- purpose CPU. An improvement is also observed with se- quential access memories, where we have observed reduc- tions of up to 99% in the required low-level memory size.

Reducing Cache Hierarchy Energy Consumption by Predicting Forwarding and Disabling Associative Sets

The first level data cache un modern processors has become a major consumer of energy due to its increasing size and high frequency access rate. In order to reduce this high energy con sumption, we propose in this paper a straightforward filtering technique based on a highly accurate forwarding predictor. Specifically, a simple structure predicts whether a load instruction will obtain its corresponding data via forwarding from the load-store structure -thus avoiding the data cache access - or if it will be provided by the data cache. This mechanism manages to reduce the data cache energy consumption by an average of 21.5% with a negligible performance penalty of less than 0.1%. Furthermore, in this paper we focus on the cache static energy consumption too by disabling a portin of sets of the L2 associative cache. Overall, when merging both proposals, the combined L1 and L2 total energy consumption is reduced by an average of 29.2% with a performance penalty of just 0.25%. Keywords: Energy consumption; filtering; forwarding predictor; cache hierarchy

De proprietate latini sermonis. Festo: De verborum significatione. Terencio Varron: De lingua latina ; Analogia

Sign.: a4, b-i6, k4, l-r6, s3

Officia propria sanctorum Valentinae dioecesis ex apostolica concessione et auctoritate summorum pontificum... : pars verna

Sign.: []2, A-I4, L1

Blondi Flauii... De Roma instaurata libri tres... ; Blondi Flauii... De Italia illustrata : opus tum propter historiarum cognitionem, tum propter locos descriptionem... [R. Volaterrani Comentario[rum] Urbano[rum]... ]

Colofón en h. 234 v.

Providencia

Contiene en [1]: "Arancel de los derechos, que han de percibir y cobrar los corredores de Vituallas, Aceyte, Miel, y Arroz ..."

Codigo de las costumbres maritimas de Barcelona, hasta aqui vulgarmente llamado Libro del Consulado

Sign.: [A]-K4, L1, A-Z4, 2A-2Z4

Addicion al discurso legal que se escrivio por los PP. Preposito, y diputados ... de la Real Casa Congregación del Oratorio de S. Felipe Neri ... como administradores de la obra pia fundada por Serafina Adrià en el pleyto con Don Vicente Sellent sobre la succesion de los bienes de Francisco Juan Adria ... para que se les absuelva de la demanda propuesta por el mismo Sellent ..

Precede al tít.: "Jesus, Maria, Joseph y San Felipe Neri"

Memorial ajustado hecho con citacion e intervencion de las partes ... de el pleyto que en grado de revista siguen la religion de S. Juan de Jerusalen y Frey D. Silverio Doz, Comendador de Torrente contra ... D. Joseph Martin Gil sobre que se declare que el dominio util que se dice tenian Andrès, y Ana Maria Gil en una casa Alqueria, con su almazara ... y que se le diesse la possession real, y efectiva al Comendador

Sign.: []1, A1, B-K2, L1

Laminar counterflow parallel-plate heat exchangers: Exact and approximate solutions

Multilayered, counterflow, parallel-plate heat exchangers are analyzed numerically and theoretically. The analysis, carried out for constant property fluids, considers a hydrodynamically developed laminar flow and neglects longitudinal conduction both in the fluid and in the plates. The solution for the temperature field involves eigenfunction expansions that can be solved in terms of Whittaker functions using standard symbolic algebra packages, leading to analytical expressions that provide the eigenvalues numerically. It is seen that the approximate solution obtained by retaining the first two modes in the eigenfunction expansion provides an accurate representation for the temperature away from the entrance regions, specially for long heat exchangers, thereby enabling simplified expressions for the wall and bulk temperatures, local heat-transfer rate, overall heat-transfer coefficient, and outlet bulk temperatures. The agreement between the numerical and theoretical results suggests the possibility of using the analytical solutions presented herein as benchmark problems for computational heat-transfer codes.

Guiding functional connectivity estimation by structural connectivity in MEG: an application to discrimination of mild cognitive impaired conditions

Whole brain resting state connectivity is a promising biomarker that might help to obtain an early diagnosis in many neurological diseases, such as dementia. Inferring resting-state connectivity is often based on correlations, which are sensitive to indirect connections, leading to an inaccurate representation of the real backbone of the network. The precision matrix is a better representation for whole brain connectivity, as it considers only direct connections. The network structure can be estimated using the graphical lasso (GL), which achieves sparsity through l1-regularization on the precision matrix. In this paper, we propose a structural connectivity adaptive version of the GL, where weaker anatomical connections are represented as stronger penalties on the corre- sponding functional connections. We applied beamformer source reconstruction to the resting state MEG record- ings of 81 subjects, where 29 were healthy controls, 22 were single-domain amnestic Mild Cognitive Impaired (MCI), and 30 were multiple-domain amnestic MCI. An atlas-based anatomical parcellation of 66 regions was ob- tained for each subject, and time series were assigned to each of the regions. The fiber densities between the re- gions, obtained with deterministic tractography from diffusion-weighted MRI, were used to define the anatomical connectivity. Precision matrices were obtained with the region specific time series in five different frequency bands. We compared our method with the traditional GL and a functional adaptive version of the GL, in terms of log-likelihood and classification accuracies between the three groups. We conclude that introduc- ing an anatomical prior improves the expressivity of the model and, in most cases, leads to a better classification between groups.